persist_js_state$a8d4a517-84a7-426e-a49e-482c5fd047aerunning§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$a8d4a517-84a7-426e-a49e-482c5fd047aedepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&Chas_pluto_hook_features¤body
The Likelihood Function is a Function of What? (*)
When considering the distribution $p(D|\theta)$ , is it more correct to speak about the likelihood of the model parameters $\theta$ than about the likelihood of the observed data set $D$ . And why?
persist_js_state$3e196d6a-d294-11ef-0795-41c045079251running§runtime |depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e196d6a-d294-11ef-0795-41c045079251depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&MƷhas_pluto_hook_features¤bodyپ
The Bayesian viewpoint is clearly favored in the machine learning community. (In this class, we also strongly favor the Bayesian interpretation).
persist_js_state$218d3b6e-50b6-4b98-a00c-a19dd33d2c03running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$218d3b6e-50b6-4b98-a00c-a19dd33d2c03depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤body٦
Let's plot the distributions for $X$ , $Y$ , and $Z$ .
persist_js_state$3e1b4b1c-d294-11ef-0423-9152887cc403running§runtime \depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1b4b1c-d294-11ef-0423-9152887cc403depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&jhas_pluto_hook_features¤body
Independent, Exclusive, and Exhaustive Events
It will be helpful to introduce some terms concerning special relationships between events.
Joint events
The expression $p(A,B)$ for the probability of the conjuction $A \land B$ is also called the joint probability of events $A$ and $B$ . Note that
$$p(A,B) = p(B,A)\,,$$
since $A\land B = B \land A$ . Therefore, the order of arguments in a joint probability distribution does not matter: $p(A,B,C,D) = p(C,A,D,B)$ , etc.
Independent events
Two events $A$ and $B$ are said to be independent if the probability of one event is not altered by information about the truth of the other event, i.e.,
$$p(A|B) = p(A)\,.$$
It follows that, if $A$ and $B$ are independent, then the product rule simplifies to
$$p(A,B) = p(A) p(B)\,.$$
$A$ and $B$ with given background $I$ are said to be conditionally independent for given $I$ , if
$$p(A|B,I) = p(A|I)\,.$$
In that case, the product rule simplifies to $p(A,B|I) = p(A|I) p(B|I)$ .
Mutually exclusive events
Two events $A_1$ and $A_2$ are said to be mutually exclusive ('disjoint') if they cannot be true simultaneously, i.e., if
$$p(A_1,A_2)=0 \,.$$
For mutually exclusive events, probabilities add (this follows from the sum rule), hence
$$p(A_1 + A_2) = p(A_1) + p(A_2)$$
Collectively exhaustive events
A set of events $A_1, A_2, \ldots, A_N$ is said to be collectively exhaustive if one of the statements is necessarily true, i.e., $A_1+A_2+\cdots +A_N=\mathrm{TRUE}$ , or equivalently
$$p(A_1+A_2+\cdots +A_N) = 1 \,.$$
Partitioning the universe
If a set of events $A_1, A_2, \ldots, A_n$ is both mutually exclusive and collectively exhaustive , then we say that they partition the universe . Technically, this means that
$$\sum_{n=1}^N p(A_n) = p(A_1 + \ldots + A_N) = 1$$
persist_js_state$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52crunning§runtimedepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52cdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&խhas_pluto_hook_features¤body?
Parameters for X
Normal(μ=
Parameters for Y
Normal(μ=
persist_js_state$3e1eeb14-d294-11ef-1702-f5d2cf6fe60arunning§runtime vdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1eeb14-d294-11ef-1702-f5d2cf6fe60adepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&*has_pluto_hook_features¤body
persist_js_state$922770f4-ddc8-4089-b378-f14088276b43running§runtimeCdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$922770f4-ddc8-4089-b378-f14088276b43depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&=xhas_pluto_hook_features¤body)
Inference Exercise: Which color does the ball have?persist_js_state$f8c8ba53-df36-48a6-afde-2952cbcfbe48running§runtime 'depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$f8c8ba53-df36-48a6-afde-2952cbcfbe48depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&ahas_pluto_hook_features¤body2
Does this fragment resonate with your own experience?
In this lesson we introduce Probability Theory (PT) again. As we will see in the next lessons, PT is all you need to make sense of machine learning, artificial intelligence, statistics, etc.
persist_js_state$41bee964-a0a9-4a7f-8505-54a9ee12ef0drunning§runtime npdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$41bee964-a0a9-4a7f-8505-54a9ee12ef0ddepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&kHhas_pluto_hook_features¤body
Propositional (Boolean) Logic
Define an event (or "proposition") $A$ as a statement that can be considered for its truth by a person. For instance,
$$𝐴= \texttt{``there is life on Mars''}$$
Boolean logic (or propositional logic) is a formal system of logic based on binary truth values: every proposition is either true (with assigned value $1$ ) or false (with assigned value $0$ ). It is named after George Boole, who developed the algebraic formulation of logic in the mid-19th century.
With Boolean operators ($\lor$ , $\land$ , $\implies$ , etc.), we can create and evaluate compound propositions, e.g.,
Given two events $A$ and $B$ , the conjunction (logical-and) $A \land B$ is true if-and-only-if both $A$ and $B$ are true. We write $A \land B$ also shortly as $AB$ (or use a comma as in a joint probability distribution $p(A,B)$ ).
The disjunction (logical-or) $A \lor B$ , is true if either $A$ or $B$ is true or both $A$ and $B$ are true. We write $A \lor B$ also as $A + B$ (Note that the plus-sign is here not an arithmetic operator, but rather a logical operator to process truth values.)
The denial of $A$ , i.e., the event not -A, is written as $\bar{A}$ .
Boolean logic provides the rules of inference for deductive reasoning and underpins all formal reasoning systems in mathematics and philosophy.
persist_js_state$3e1b8bf4-d294-11ef-04cc-6364e46fdd64running§runtime ]7depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1b8bf4-d294-11ef-04cc-6364e46fdd64depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& has_pluto_hook_features¤bodyw
A very practical application of this law is to get rid of a variable that we are not interested in. For instance, if $X$ and $Y \in \{y_1,y_2,\ldots,y_n\}$ are discrete variables, then
$$p(X) = \sum_{i=1}^n p(X,Y=y_i)\,.$$
persist_js_state$98fa17a6-7c8b-46e4-b32d-52db183d88f8running§runtime xdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$98fa17a6-7c8b-46e4-b32d-52db183d88f8depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&]has_pluto_hook_features¤bodyَ
Set the parameters for the distributions of $X$ and $Y$ :
persist_js_state$3e1f4f46-d294-11ef-29b8-69e546763781running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1f4f46-d294-11ef-29b8-69e546763781depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& dhas_pluto_hook_features¤body
PDF for the Product of Two Variables
For two continuous independent variables $X$ and $Y$ , with PDF's $p_x(x)$ and $p_y(y)$ , the PDF of $Z = X Y$ is given by
$$p_z(z) = \int_{-\infty}^{\infty} p_x(x) \,p_y(z/x)\, \frac{1}{|x|}\,\mathrm{d}x\,.$$
For proof, see https://en.wikipedia.org/wiki/Product_distribution .
persist_js_state$3a1d380e-df80-4727-9772-f199214cf05drunning§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3a1d380e-df80-4727-9772-f199214cf05ddepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&1has_pluto_hook_features¤bodyq
The Sum Rule (**)
Derive the general sum rule,
$$p(A + B) = p(A) + p(B) - p(A,B)$$
from the elementary sum rule $p(A) + p(\bar A) = 1$ and the product rule.
persist_js_state$3e1e5a5a-d294-11ef-2fdf-efee4eb1a0f2running§runtime %depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1e5a5a-d294-11ef-2fdf-efee4eb1a0f2depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&
?has_pluto_hook_features¤body
The second central moment, also known as variance of $x$ is defined as
$$\Sigma_x \triangleq \mathbb{E} \left[(x-\mu_x)(x-\mu_x)^T \right]$$
persist_js_state$3e1bdd02-d294-11ef-19e8-2f44eccf58afrunning§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1bdd02-d294-11ef-19e8-2f44eccf58afdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤body
This last formula is called Bayes rule , named after its inventor Thomas Bayes (1701-1761). While Bayes rule is always true, a particularly useful application occurs when $D$ refers to an observed data set and $\theta$ is set of unobserved model parameters. In that case,
the prior probability $p(\theta)$ represents our state-of-knowledge about proper values for $\theta$ , before seeing the data $D$ .
the posterior probability $p(\theta|D)$ represents our state-of-knowledge about $\theta$ after we have seen the data.
persist_js_state$fea8ae4c-8ef9-4b74-ad13-1314afef97derunning§runtime Fdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$fea8ae4c-8ef9-4b74-ad13-1314afef97dedepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤body
True and False Events
In probability theory, events that are certainly true or certainly false are treated as special cases of general events, and they correspond to probabilities of $1$ and $0$ , respectively.
Let $\Omega$ be the sample space, i.e., the set of all possible outcomes of an experiment. Then:
The true event corresponds to the entire sample space $\Omega$ .
It always happens, regardless of the outcome.
For example, for the outcome of a throw of a dice, the sample space is $\Omega = \{1,2,3,4,5,6\}$ .
Its probability is
$$p(\Omega) = 1\,.$$
The false event corresponds to the empty set $\emptyset$ .
$$p(\emptyset) = 0 \,.$$
persist_js_state$db73766d-643c-41d7-a1eb-f376c657f860running§runtime Xdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$db73766d-643c-41d7-a1eb-f376c657f860depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&
has_pluto_hook_features¤body3
If the transformation $y=h(x)$ is not invertible, then $x=g(y)$ does not exist. In that case, you can still work out the transformation by equating equivalent probability masses in the two domains.
persist_js_state$3b1b0869-b815-4697-9dba-3c4b4cb5ac47running§runtime jdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3b1b0869-b815-4697-9dba-3c4b4cb5ac47depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&bhas_pluto_hook_features¤body D
Click for the solution
$$\begin{align}
p\left( A + B \right) &\underset{\mathrm{bool}}{=} p\left( \overline {\bar A \bar B } \right) \\
&\underset{\mathrm{sum}}{=} 1 - p\left( \bar{A} \bar{B} \right) \\
&\underset{\mathrm{prod}}{=} 1 - p\left( \bar{A} |\bar{B} \right) p\left(\bar{B} \right) \\
&\underset{\mathrm{sum}}{=} 1 - \left( 1 - p\left(A|\bar B \right) \right) \left( 1 - p\left( B \right) \right) \\
&= p(B) + \left( {1 - p\left( B \right)} \right)p\left( {A|\bar B } \right) \\
&\underset{\mathrm{prod}}{=} p(B) + \left( 1 - p\left( B \right) \right) p\left( \bar{B} |A \right) \frac{ p\left( A \right) }{ p\left(\bar{B}\right)} \\
&\underset{\mathrm{sum}}{=} p(B) + p\left(\bar{B} |A \right) p\left( A \right) \\
&\underset{\mathrm{sum}}{=} p(B) + \left( 1 - p\left( {B|A} \right) \right) p\left( A \right) \\
&\underset{\mathrm{sum}}{=} p\left( A \right) + p(B) - p\left( A,B \right)
\end{align}$$
Note that, aside from the first boolean rewrite, everything follows straight application of sum and product rules.
persist_js_state$a8046381-ff11-40af-ae2b-078d71c586e7running§runtimeWdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$a8046381-ff11-40af-ae2b-078d71c586e7depends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneeresultlast_run_timestampAڹ&|`Whas_pluto_hook_features¤body0.060126582278481persist_js_state$3e1d20e0-d294-11ef-2044-e1fe6590a600running§runtime :Ÿdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1d20e0-d294-11ef-2044-e1fe6590a600depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&
has_pluto_hook_features¤body
Note
The (discrete) sampling distribution is a valid probability distribution.
However, the likelihood function $L(\theta)$ clearly isn't, since $\int_0^1 L(\theta) \mathrm{d}\theta = 0.5 \neq 1$ .
persist_js_state$fc733d61-fd0f-4a13-9afc-4505ac0253dfrunning§runtimeZTdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$fc733d61-fd0f-4a13-9afc-4505ac0253dfdepends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneelast_run_timestampAڹ&)has_pluto_hook_features¤bodypersist_js_state$3e1c60ba-d294-11ef-3a01-cf9e97512857running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1c60ba-d294-11ef-3a01-cf9e97512857depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& 'ķhas_pluto_hook_features¤bodyK
In general, $p(D|\theta)$ is just a function of the two variables $D$ and $\theta$ . We distinguish two interpretations of this function, depending on which variable is observed (or given by other means).
persist_js_state$3e1bf116-d294-11ef-148b-f7a1ca3f3badrunning§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1bf116-d294-11ef-148b-f7a1ca3f3baddepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& D5has_pluto_hook_features¤bodyو
Bayes rule tells us how to update our knowledge about model parameters when facing new data. Hence,
persist_js_state$3e1fa04a-d294-11ef-00c3-a51d1aaa5553running§runtime 9˸depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1fa04a-d294-11ef-00c3-a51d1aaa5553depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&Ʒhas_pluto_hook_features¤bodyx
If $x$ is defined on a continuous domain, and $p_x(x)$ is a probability density function, then probability mass is represented by the area under a (density) curve. In that case,
$$p_y(y) = p_x(g(y)) g^\prime(y)\,,$$
which is also known as the Change-of-Variable theorem .
persist_js_state$9b85a92b-c56b-48a3-97c1-6b1882e33a22running§runtimeYdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$9b85a92b-c56b-48a3-97c1-6b1882e33a22depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&Dyhas_pluto_hook_features¤bodyٟ
Show solution?
persist_js_state$079157c9-5d97-4dbc-8c47-afa8b661db06running§runtimeܸdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$079157c9-5d97-4dbc-8c47-afa8b661db06depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&3ʷhas_pluto_hook_features¤bodyق
\[p(y|\theta=0.50) = \begin{cases} 0.50 & \text{if }y=0 \\ 0.50 & \text{if } y=1 \end{cases}\]
persist_js_state$3e1c70be-d294-11ef-14ed-0d46515541c5running§runtime gdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1c70be-d294-11ef-14ed-0d46515541c5depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& Khas_pluto_hook_features¤body
The sampling distribution (a.k.a. the data-generating distribution)
$$p(D|\theta=\theta_0)$$
(which is a function of $D$ only) describes a probability distribution for data $D$ , assuming that it is generated by the given model with parameters fixed at $\theta = \theta_0$ .
persist_js_state$3e1bffec-d294-11ef-2a49-9ff0f6331addrunning§runtime M depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1bffec-d294-11ef-2a49-9ff0f6331adddepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& fhas_pluto_hook_features¤body
Bayes Rule Nomenclature
Some nomenclature associated with Bayes rule:
$$\underbrace{p(\theta | D)}_{\text{posterior}} = \frac{\overbrace{p(D|\theta)}^{\text{likelihood}} \times \overbrace{p(\theta)}^{\text{prior}}}{\underbrace{p(D)}_{\text{evidence}}}$$
persist_js_state$3e198ba6-d294-11ef-3fe7-d70bf4833fa6running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e198ba6-d294-11ef-3fe7-d70bf4833fa6depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤bodyٮ
In this class, we aim to turn on the headlights and illuminate the elegance and power of the Bayesian approach to information processing.
persist_js_state$3e1a8eca-d294-11ef-1ef0-c15b24d05990running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1a8eca-d294-11ef-1ef0-c15b24d05990depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&ɷhas_pluto_hook_features¤body
If $X$ is continuously valued, then $p(X=x)$ is a probability density function (PDF) with $p(X=x)\ge 0$ and normalization $\int_x p(x)\mathrm{d}x=1$ .
Note that if $X$ is continuously valued, then the value of $p(x)$ is not necessarily $\le 1$ . E.g., a uniform distribution on the continuous domain $[0,.5]$ has value $p(x) = 2$ over its domain.
persist_js_state$3e1f68fa-d294-11ef-31b2-e7670da8c08crunning§runtime 1depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1f68fa-d294-11ef-31b2-e7670da8c08cdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&F|has_pluto_hook_features¤body(
persist_js_state$d0f672bd-74c9-4be1-b5ea-8d4040edb5dfrunning§runtime$depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$d0f672bd-74c9-4be1-b5ea-8d4040edb5dfdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&f1has_pluto_hook_features¤body
persist_js_state$3e1c51e2-d294-11ef-2c6d-d32a98308c6frunning§runtime Pdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1c51e2-d294-11ef-2c6d-d32a98308c6fdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ& ڷhas_pluto_hook_features¤body
The Likelihood Function vs the Sampling Distribution
Consider a distribution $p(D|\theta)$ , where $D$ relates to variables that are observed (i.e., a "data set") and $\theta$ are model parameters.
persist_js_state$3e1a7c8e-d294-11ef-1f97-55e608d49141running§runtime (Ÿdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1a7c8e-d294-11ef-1f97-55e608d49141depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&{has_pluto_hook_features¤body]
If $X$ is a discretely valued variable, then $p(X=x)$ is a probability mass function (PMF) with $0\le p(X=x)\le 1$ and normalization $\sum_x p(x) =1$ .
persist_js_state$b305a905-06c2-4a15-8042-72ef6375720frunning§runtimedgdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$b305a905-06c2-4a15-8042-72ef6375720fdepends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneelast_run_timestampAڹ&Phas_pluto_hook_features¤bodypersist_js_state$3e1d6d00-d294-11ef-1081-e11b8397eb91running§runtime !depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1d6d00-d294-11ef-1081-e11b8397eb91depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&pbhas_pluto_hook_features¤body%
Problem
Given a disease $D \in \{0, 1\}$ with prevalence (overall occurence percentage) of $T \in \{0, 1\}$ with sensitivity (true positive rate) of
The percentages are interactive! Click and drag to change the values.
persist_js_state$3e191b6c-d294-11ef-3174-d1b4b36e252brunning§runtime udepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e191b6c-d294-11ef-3174-d1b4b36e252bdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&ܷhas_pluto_hook_features¤bodyg
Examples
$$p(\,\texttt{future-observations}\,|\,\texttt{past-observations}\,)$$
$$p(\,x\texttt{-belongs-to-class-}k \,|\,x\,)$$
$$p(\,\texttt{model-parameters} \,|\,\texttt{new-observation},\,\texttt{past-observations}\,)$$
persist_js_state$4a81342c-17c7-4eb9-933b-edb98df7b9c4running§runtime][depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$4a81342c-17c7-4eb9-933b-edb98df7b9c4depends_on_disabled_cells¦queued¤logsoutputmimetext/plainrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤body"n (generic function with 1 method)persist_js_state$9f4125a2-d5d2-4acf-8bad-82f94af230e8running§runtime>depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$9f4125a2-d5d2-4acf-8bad-82f94af230e8depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&ߕhas_pluto_hook_features¤body
Preface
"As an undergraduate, I always found the subject of statistics to be rather mysterious. This topic wasn’t entirely new to me, as we had been taught a little bit about probability earlier at high school; for example, I was already familiar with the binomial, Poisson and normal distributions. Most of this made sense, but only seemed to relate to things like rolling dice, flipping coins, shuffling cards and so on. However, having aspirations of becoming a scientist, what I really wanted to know was how to analyse experimental data. Thus, I eagerly looked forward to the lectures on statistics. Sadly, they were a great disappointment. Although many of the tests and procedures expounded were intuitively reasonable, there was something deeply unsatisfactory about the whole affair: there didn’t seem to be any underlying basic principles! Hence, the course on ‘probability and statistics’ had led to an unfortunate dichotomy: probability made sense, but was just a game; statistics was important, but it was a bewildering collection of tests with little obvious rhyme or reason. While not happy with this situation, I decided to put aside the subject and concentrate on real science. After all, the predicament was just a reflection of my own inadequacies and I’d just have to work at it when the time came to really analyse my data.
The story above is not just my own, but is the all too common experience of many scientists. Fortunately, it doesn’t have to be like this. What we were not told in our undergraduate lectures is that there is an alternative approach to the whole subject of data analysis which uses only probability theory. In one sense, it makes the topic of statistics entirely superfluous. In another, it provides the logical justification for many of the prevalent statistical tests and procedures, making explicit the conditions and approximations implicitly assumed in their use."
D.S. Sivia
persist_js_state$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafrunning§runtime ^depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&1
has_pluto_hook_features¤body|
Probability Theory as Rational Reasoning
persist_js_state$3e027ede-4ac1-4521-a026-dc00bfca4adfrunning§runtime%depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e027ede-4ac1-4521-a026-dc00bfca4adfdepends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&?=6has_pluto_hook_features¤body
Inference Exercise: Causality?persist_js_state$3e1889b8-d294-11ef-17bb-496655fbd618running§runtime 'depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1889b8-d294-11ef-17bb-496655fbd618depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&ķhas_pluto_hook_features¤body
The Design of Probability Theory
In the real world, we are rarely completely certain of something. Instead of a truth value for the proposition $A$ , we can attach a degree-of-belief $p(A)$ that represents how likely we think $A$ is.
Now consider the truth value of the proposition
$$𝐴= \texttt{``there is life on Mars''}$$
with
$$I = \texttt{``All known life forms require water''}$$
as background information. Now, assume that a new piece of information
$$B = \texttt{``There is water on Mars''}$$
becomes available, how should our degree of belief in event $A$ be affected if we were rational ?
persist_js_state$6d07be25-53d0-46b9-b197-a3680d830952running§runtime ٸdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$6d07be25-53d0-46b9-b197-a3680d830952depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&Ehas_pluto_hook_features¤body o
Click for the solution
Define $A = \begin{pmatrix} I & I \end{pmatrix}$ and $w = \begin{pmatrix} x \\ y \end{pmatrix}$ , where $I$ is the identity matrix. Then
$$z = A w\,. $$
Let $\mathbb{E}[\cdot]$ refer to the expectation operator. Now apply the formula for the mean and variance of a variable after a linear transformation:
$$\mathbb{E}[z] = \mathbb{E}[Aw] = \mathbb{E}[x+y] = \mathbb{E}[x] + \mathbb{E}[y] \,. $$
For the covariance matrix, first note that
$$\begin{align}
\Sigma_w &= \mathbb{E} \bigg[ \begin{pmatrix} x - \mu_x \\ y- \mu_y\end{pmatrix} \begin{pmatrix} x - \mu_x \\ y- \mu_y\end{pmatrix}^T \bigg] \\
&= \begin{pmatrix} \Sigma_x & \Sigma_{xy} \\ \Sigma_{yx} & \Sigma_y\end{pmatrix}
\end{align} $$
and we note that $\Sigma_{yx} = \Sigma_{xy}^T$ . Then, $\Sigma_z$ evaluates to
$$\begin{align}
\Sigma_z &= A \Sigma_w A^T \\
&= \begin{pmatrix} I & I \end{pmatrix} \begin{pmatrix} \Sigma_x & \Sigma_{xy} \\ \Sigma_{yx} & \Sigma_y\end{pmatrix} \begin{pmatrix}I \\ I \end{pmatrix} \\
&= \Sigma_x + \Sigma_y + \Sigma_{xy} + \Sigma_{yx} \,.
\end{align}$$
persist_js_state$5f377237-d9a5-4778-aa4d-1c6ce109b705running§runtime depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$5f377237-d9a5-4778-aa4d-1c6ce109b705depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&has_pluto_hook_features¤body
Apples and Oranges
Box 1 contains 8 apples and 4 oranges. Box 2 contains 10 apples and 2 oranges. Boxes are chosen with equal probability. You pick a box and then select a fruit from that box.
(a) (*) What is the probability of choosing an apple?
(b) (**) If an apple is chosen, what is the probability that it came from box 1?
persist_js_state$d3b003c6-70ca-419f-a343-e35b266323f3running§runtime Qdepends_on_skipped_cellsµpublished_object_keyserrored§cell_id$d3b003c6-70ca-419f-a343-e35b266323f3depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&cihas_pluto_hook_features¤bodyO
Click for the solution
Yes, it’s more correct to speak about the likelihood of the model parameters, not of the observed data set. Once $D$ has been observed, it is no longer a random variable; it’s just a fixed outcome. What varies is $\theta$ , so $L(\theta) = p(D|\theta)$ is a function of the parameters, not of the data.
Saying “likelihood of the data” is misleading because it confuses likelihood with the sampling distribution $p(D|\theta)$ seen as a function of $D$ (where $\theta$ is fixed). The latter is a probability distribution over possible data sets before observing them.
persist_js_state$3e1fb370-d294-11ef-1fb6-63a41a024691running§runtime `depends_on_skipped_cellsµpublished_object_keyserrored§cell_id$3e1fb370-d294-11ef-1fb6-63a41a024691depends_on_disabled_cells¦queued¤logsoutputmimetext/htmlrootassigneelast_run_timestampAڹ&`chas_pluto_hook_features¤body
Let $p_x(x) = \mathcal{N}(x|\mu,\sigma^2)$ where
$$\mathcal{N}(x|\mu,\sigma^2) \triangleq \frac{1}{\sigma \sqrt{2\pi}} \,\exp\left(-\frac{(x-\mu)^2}{2 \sigma^2}\right)$$
is a Gaussian distribution . Let $y = \frac{x-\mu}{\sigma}$ . Evaluate $p_y(y)$ as a Gaussian distribution.
persist_js_state±published_objectsjulia_versionv1.13.3cell_dependencies $3e1bba8e-d294-11ef-1f61-295af16078ceprecedence_heuristic cell_id$3e1bba8e-d294-11ef-1f61-295af16078cedownstream_cells_mapupstream_cells_map@md_strgetindex$3e1eba72-d294-11ef-2f53-b56f1862fcbbprecedence_heuristic cell_id$3e1eba72-d294-11ef-2f53-b56f1862fcbbdownstream_cells_mapupstream_cells_map@md_strgetindex$9b9be452-9681-43e8-bb09-cc8728df384fprecedence_heuristic cell_id$9b9be452-9681-43e8-bb09-cc8728df384fdownstream_cells_mapupstream_cells_map@md_strgetindex$3e1f8e48-d294-11ef-0f8a-b58294a8543dprecedence_heuristic cell_id$3e1f8e48-d294-11ef-0f8a-b58294a8543ddownstream_cells_mapupstream_cells_map@md_strhide_proofgetindex$5b681e41-ad14-4c58-8ea0-4b6d85885c51precedence_heuristic cell_id$5b681e41-ad14-4c58-8ea0-4b6d85885c51downstream_cells_mapupstream_cells_map@md_strgetindex$42b47af6-b850-4987-a2d7-805a2cb64e43precedence_heuristiccell_id$42b47af6-b850-4987-a2d7-805a2cb64e43downstream_cells_map@mdx$079157c9-5d97-4dbc-8c47-afa8b661db06$2156f96e-eebe-4190-8ce9-c76825c6da71upstream_cells_map$726e7e9f-3d68-4d02-9954-1edd98b74fdcprecedence_heuristic cell_id$726e7e9f-3d68-4d02-9954-1edd98b74fdcdownstream_cells_mapupstream_cells_map@md_strgetindexkeyconcept$3e18e4bc-d294-11ef-38bc-cb97cb4e0963precedence_heuristic cell_id$3e18e4bc-d294-11ef-38bc-cb97cb4e0963downstream_cells_mapupstream_cells_map@md_strgetindexkeyconcept$e3157dc0-5a64-4479-a37a-40fe25cccc07precedence_heuristic cell_id$e3157dc0-5a64-4479-a37a-40fe25cccc07downstream_cells_mapupstream_cells_mapcode_example$3e1c806a-d294-11ef-1fad-17e5625279f7precedence_heuristic cell_id$3e1c806a-d294-11ef-1fad-17e5625279f7downstream_cells_mapupstream_cells_map@md_strgetindex$3e1ea442-d294-11ef-1364-8dd9986325f7precedence_heuristic cell_id$3e1ea442-d294-11ef-1364-8dd9986325f7downstream_cells_mapupstream_cells_map@md_strgetindex$3e1c4224-d294-11ef-2707-49470aaae6ebprecedence_heuristic cell_id$3e1c4224-d294-11ef-2707-49470aaae6ebdownstream_cells_mapupstream_cells_map@md_strgetindex$3e1f225a-d294-11ef-04c6-f3ca018ab286precedence_heuristic cell_id$3e1f225a-d294-11ef-04c6-f3ca018ab286downstream_cells_mapupstream_cells_mapcode_example@md_strgetindex$5613e9b7-ff0d-435a-9de6-aaf293ebf592precedence_heuristic cell_id$5613e9b7-ff0d-435a-9de6-aaf293ebf592downstream_cells_mapupstream_cells_maphide_solution@md_strgetindex$3e1c1e3e-d294-11ef-0955-bdf9d0ba3c53precedence_heuristic cell_id$3e1c1e3e-d294-11ef-0955-bdf9d0ba3c53downstream_cells_mapupstream_cells_map@md_strgetindex$9b92fc89-2036-4525-979b-d296ab29329cprecedence_heuristic cell_id$9b92fc89-2036-4525-979b-d296ab29329cdownstream_cells_mapupstream_cells_map@md_strgetindex$eeb9a1f5-b857-4843-920b-2e4a9656f66bprecedence_heuristiccell_id$eeb9a1f5-b857-4843-920b-2e4a9656f66bdownstream_cells_mapLaTeXStrings$8a7dd8b7-5faf-4091-8451-9769f842accb$c0ea3253-a06b-426c-91a3-a6dd33e42779Plotsupstream_cells_map$5377c5a4-77c4-4fa7-9f84-0c511e3bf708precedence_heuristic cell_id$5377c5a4-77c4-4fa7-9f84-0c511e3bf708downstream_cells_mapupstream_cells_map@md_strhide_proofgetindex$3e1e7742-d294-11ef-1204-f9be24da07abprecedence_heuristic cell_id$3e1e7742-d294-11ef-1204-f9be24da07abdownstream_cells_mapupstream_cells_map@md_strgetindex$ff9142ba-3a85-48cf-8b78-07e0b554e280precedence_heuristic cell_id$ff9142ba-3a85-48cf-8b78-07e0b554e280downstream_cells_mapupstream_cells_map@md_strgetindex$ea803646-b18e-4f13-ab6a-fc5080d44e92precedence_heuristic cell_id$ea803646-b18e-4f13-ab6a-fc5080d44e92downstream_cells_mapupstream_cells_mapchallenge_solution$3e192ef4-d294-11ef-1fc4-87175eeec5ebprecedence_heuristic cell_id$3e192ef4-d294-11ef-1fc4-87175eeec5ebdownstream_cells_mapupstream_cells_map@md_strgetindex$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0precedence_heuristic cell_id$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0downstream_cells_mapupstream_cells_map@md_strgetindex$317707a3-9ef1-4c67-b451-6adcfcff50f0precedence_heuristic cell_id$317707a3-9ef1-4c67-b451-6adcfcff50f0downstream_cells_mapupstream_cells_maphide_solution@md_strgetindex$8a7dd8b7-5faf-4091-8451-9769f842accbprecedence_heuristic cell_id$8a7dd8b7-5faf-4091-8451-9769f842accbdownstream_cells_mapupstream_cells_mapLaTeXStrings.latexstring:f$fc733d61-fd0f-4a13-9afc-4505ac0253dfLaTeXStrings$eeb9a1f5-b857-4843-920b-2e4a9656f66bplotnothingscatter!θ$fc733d61-fd0f-4a13-9afc-4505ac0253df@L_str$99d9099f-4908-4bb3-8d59-da9cb69af04cprecedence_heuristic cell_id$99d9099f-4908-4bb3-8d59-da9cb69af04cdownstream_cells_mapupstream_cells_maphint@md_strgetindex$dd11e93a-3dad-4e97-8642-fb70edfa6aaeprecedence_heuristic cell_id$dd11e93a-3dad-4e97-8642-fb70edfa6aaedownstream_cells_mapupstream_cells_map@md_strgetindex$3e1de32c-d294-11ef-1f63-f190c8361404precedence_heuristic cell_id$3e1de32c-d294-11ef-1f63-f190c8361404downstream_cells_mapupstream_cells_map@md_strgetindex$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4precedence_heuristic cell_id$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4downstream_cells_mapupstream_cells_map@md_strgetindex$3e1a69f4-d294-11ef-103e-efc47025fb8fprecedence_heuristic cell_id$3e1a69f4-d294-11ef-103e-efc47025fb8fdownstream_cells_mapupstream_cells_map@md_strgetindex$3e198336-d294-11ef-26fd-03cd15876486precedence_heuristic cell_id$3e198336-d294-11ef-26fd-03cd15876486downstream_cells_mapupstream_cells_map@md_strgetindex$6fafc778-c2c8-4e23-8d2e-6dc02e49e935precedence_heuristic cell_id$6fafc778-c2c8-4e23-8d2e-6dc02e49e935downstream_cells_mapupstream_cells_map@md_strgetindex$5394e37c-ae00-4042-8ada-3bbf32fbca9eprecedence_heuristiccell_id$5394e37c-ae00-4042-8ada-3bbf32fbca9edownstream_cells_mapDistributionsupstream_cells_map$3e1ab104-d294-11ef-1a98-412946949fbaprecedence_heuristic cell_id$3e1ab104-d294-11ef-1a98-412946949fbadownstream_cells_mapupstream_cells_map@md_strgetindexHTML$c0ea3253-a06b-426c-91a3-a6dd33e42779precedence_heuristic cell_id$c0ea3253-a06b-426c-91a3-a6dd33e42779downstream_cells_mapupstream_cells_mapplot!@L_strx$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52cLaTeXStrings.latexstringpdfy$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52cLaTeXStrings$eeb9a1f5-b857-4843-920b-2e4a9656f66bplotz$34f1f3cb-9346-4bd3-ae9f-a25b4d5de781range$b176ceae-884e-4460-9f66-020c1ac447f1precedence_heuristic cell_id$b176ceae-884e-4460-9f66-020c1ac447f1downstream_cells_mapupstream_cells_map@md_strgetindex$d2202628-e4f9-4289-b48e-23b5a0073f94precedence_heuristic cell_id$d2202628-e4f9-4289-b48e-23b5a0073f94downstream_cells_mapupstream_cells_map@md_strhide_proofgetindex$b7445b9b-7fbb-4560-b947-a23af0fcf101precedence_heuristic cell_id$b7445b9b-7fbb-4560-b947-a23af0fcf101downstream_cells_mapupstream_cells_mapθ_bond$fc733d61-fd0f-4a13-9afc-4505ac0253df@md_strgetindex$bcb4be20-0439-4809-a166-8c50b6b9206bprecedence_heuristic cell_id$bcb4be20-0439-4809-a166-8c50b6b9206bdownstream_cells_mapupstream_cells_mapPlutoUI.TableOfContentsPlutoUI$50bdc2fe-f48d-4c4e-8b4e-170782681366precedence_heuristic cell_id$50bdc2fe-f48d-4c4e-8b4e-170782681366downstream_cells_mapupstream_cells_map@md_strhide_proofgetindex$fc3151f9-e143-4e31-b7b7-3f25b4fe9dabprecedence_heuristic cell_id$fc3151f9-e143-4e31-b7b7-3f25b4fe9dabdownstream_cells_mapupstream_cells_map@md_strgetindex$1cb8b2c4-e1ae-4973-ba53-fc6c7fe1f37aprecedence_heuristic cell_id$1cb8b2c4-e1ae-4973-ba53-fc6c7fe1f37adownstream_cells_mapupstream_cells_map@md_strgetindex$d93f73d4-2783-4777-b0ce-cdc0444cb300precedence_heuristic cell_id$d93f73d4-2783-4777-b0ce-cdc0444cb300downstream_cells_mapupstream_cells_mapθ_bond$fc733d61-fd0f-4a13-9afc-4505ac0253df@md_strgetindex$3e1803d0-d294-11ef-0304-df2b9b698cd1precedence_heuristic cell_id$3e1803d0-d294-11ef-0304-df2b9b698cd1downstream_cells_mapupstream_cells_map@md_strgetindex$3e18d2ea-d294-11ef-35e9-2332dd31dbf0precedence_heuristic cell_id$3e18d2ea-d294-11ef-35e9-2332dd31dbf0downstream_cells_mapupstream_cells_map@md_strgetindex$2156f96e-eebe-4190-8ce9-c76825c6da71precedence_heuristic cell_id$2156f96e-eebe-4190-8ce9-c76825c6da71downstream_cells_map#462#resultupstream_cells_mapHypertextLiteralHypertextLiteral.content- MarkdownLiteral.CMParsedRenderer@mdx$42b47af6-b850-4987-a2d7-805a2cb64e43show_disease_diagnosis_solution$9b85a92b-c56b-48a3-97c1-6b1882e33a22prevalence$4f6dd225-c64d-4b76-b075-0bf71c863b5aMarkdownLiteraln$4a81342c-17c7-4eb9-933b-edb98df7b9c4HypertextLiteral.Resultresult$a8046381-ff11-40af-ae2b-078d71c586e7sensitivity$4f6dd225-c64d-4b76-b075-0bf71c863b5aHypertextLiteral.Bypassspecificity$4f6dd225-c64d-4b76-b075-0bf71c863b5a$ef264651-854e-4374-8ea8-5476c85150c4precedence_heuristic cell_id$ef264651-854e-4374-8ea8-5476c85150c4downstream_cells_mapupstream_cells_map@md_strgetindex$91dd40f0-c373-48b3-b83b-6e8df2c43e5aprecedence_heuristic cell_id$91dd40f0-c373-48b3-b83b-6e8df2c43e5adownstream_cells_mapupstream_cells_maphide_solution@md_strgetindex$3e1babca-d294-11ef-37c1-cd821a6488b2precedence_heuristic cell_id$3e1babca-d294-11ef-37c1-cd821a6488b2downstream_cells_mapupstream_cells_map@md_strgetindex$58f70d3e-4b64-414e-b560-327be2a0c4c2precedence_heuristic cell_id$58f70d3e-4b64-414e-b560-327be2a0c4c2downstream_cells_mapupstream_cells_mapexercise_statement$34f1f3cb-9346-4bd3-ae9f-a25b4d5de781precedence_heuristic cell_id$34f1f3cb-9346-4bd3-ae9f-a25b4d5de781downstream_cells_mapz$c0ea3253-a06b-426c-91a3-a6dd33e42779upstream_cells_map^Normal+y$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52csqrt√x$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52c$3e18c25c-d294-11ef-11bc-a93c2572b107precedence_heuristic cell_id$3e18c25c-d294-11ef-11bc-a93c2572b107downstream_cells_mapupstream_cells_map@md_strgetindex$03692f4d-0daf-4dfc-a7ff-6b954326e4d0precedence_heuristic cell_id$03692f4d-0daf-4dfc-a7ff-6b954326e4d0downstream_cells_mapupstream_cells_mapexercises$3e1e4dda-d294-11ef-33b7-4bbe3300ca22precedence_heuristic cell_id$3e1e4dda-d294-11ef-33b7-4bbe3300ca22downstream_cells_mapupstream_cells_map@md_strgetindex$3e1d33c8-d294-11ef-0a08-bdc419949925precedence_heuristic cell_id$3e1d33c8-d294-11ef-0a08-bdc419949925downstream_cells_mapupstream_cells_map@md_strgetindex$3e194ef2-d294-11ef-3b38-1ddc3063ff35precedence_heuristic cell_id$3e194ef2-d294-11ef-3b38-1ddc3063ff35downstream_cells_mapupstream_cells_map@md_strgetindex$16c2eb59-16b8-4347-9aab-6e4b99016c79precedence_heuristic cell_id$16c2eb59-16b8-4347-9aab-6e4b99016c79downstream_cells_mapupstream_cells_map@md_strgetindexkeyconcept$3e1906ea-d294-11ef-236e-c966a9474170precedence_heuristic cell_id$3e1906ea-d294-11ef-236e-c966a9474170downstream_cells_mapupstream_cells_map@md_strgetindex$3e1bcb00-d294-11ef-2795-bd225bd00496precedence_heuristic cell_id$3e1bcb00-d294-11ef-2795-bd225bd00496downstream_cells_mapupstream_cells_map@md_strgetindexHTML$66ebe33c-8360-4938-9b51-625e5bed176cprecedence_heuristic cell_id$66ebe33c-8360-4938-9b51-625e5bed176cdownstream_cells_mapupstream_cells_maphide_solution@md_strgetindex$3e1e9224-d294-11ef-38b3-137c2be22400precedence_heuristic cell_id$3e1e9224-d294-11ef-38b3-137c2be22400downstream_cells_mapupstream_cells_map@md_strgetindex$3e17df5e-d294-11ef-38c7-f573724871d8precedence_heuristic cell_id$3e17df5e-d294-11ef-38c7-f573724871d8downstream_cells_mapupstream_cells_maptitle$70d79732-0f55-40ba-929d-fba431318848precedence_heuristic cell_id$70d79732-0f55-40ba-929d-fba431318848downstream_cells_mapupstream_cells_map@md_strgetindex$3e1964b4-d294-11ef-373d-712257fc130fprecedence_heuristic cell_id$3e1964b4-d294-11ef-373d-712257fc130fdownstream_cells_mapupstream_cells_map@md_strgetindex$3e1c9184-d294-11ef-3e35-5393d97fbc44precedence_heuristic cell_id$3e1c9184-d294-11ef-3e35-5393d97fbc44downstream_cells_mapupstream_cells_map@md_strgetindex$3e18f18c-d294-11ef-33e4-b7f9495e0508precedence_heuristic cell_id$3e18f18c-d294-11ef-33e4-b7f9495e0508downstream_cells_mapupstream_cells_map@md_strgetindex$3e1b5c9c-d294-11ef-137f-d75b3731eae4precedence_heuristic cell_id$3e1b5c9c-d294-11ef-137f-d75b3731eae4downstream_cells_mapupstream_cells_map@md_strgetindex$4edf38ab-a940-4ab0-be22-fa95cf571146precedence_heuristic cell_id$4edf38ab-a940-4ab0-be22-fa95cf571146downstream_cells_mapupstream_cells_map@md_strgetindex$a66ab9df-897c-42e5-8b0f-c520ceaffa23precedence_heuristiccell_id$a66ab9df-897c-42e5-8b0f-c520ceaffa23downstream_cells_mapPrintfupstream_cells_map$3e19e95a-d294-11ef-3da4-6d23922a5150precedence_heuristic cell_id$3e19e95a-d294-11ef-3da4-6d23922a5150downstream_cells_mapupstream_cells_map@md_strgetindex$3e19436c-d294-11ef-11c5-f9914f7a3a57precedence_heuristic cell_id$3e19436c-d294-11ef-11c5-f9914f7a3a57downstream_cells_mapupstream_cells_map@md_strgetindex$3e1b05ee-d294-11ef-33de-efed64d01c0dprecedence_heuristic cell_id$3e1b05ee-d294-11ef-33de-efed64d01c0ddownstream_cells_mapupstream_cells_map@md_strgetindexkeyconcept$4f6dd225-c64d-4b76-b075-0bf71c863b5aprecedence_heuristic cell_id$4f6dd225-c64d-4b76-b075-0bf71c863b5adownstream_cells_mapprevalence_bond$3e185ab0-d294-11ef-3f7d-9bd465518274$3e1d6d00-d294-11ef-1081-e11b8397eb91sensitivity_bond$3e185ab0-d294-11ef-3f7d-9bd465518274$3e1d6d00-d294-11ef-1081-e11b8397eb91specificity$2156f96e-eebe-4190-8ce9-c76825c6da71$a8046381-ff11-40af-ae2b-078d71c586e7sensitivity$2156f96e-eebe-4190-8ce9-c76825c6da71$a8046381-ff11-40af-ae2b-078d71c586e7specificity_bond$3e185ab0-d294-11ef-3f7d-9bd465518274$3e1d6d00-d294-11ef-1081-e11b8397eb91prevalence$2156f96e-eebe-4190-8ce9-c76825c6da71$a8046381-ff11-40af-ae2b-078d71c586e7upstream_cells_mapBaseScrubbable:@bindBase.getPlutoRunnerCore.applicableCorePlutoRunner.create_bond$61713e1c-8e37-45d9-9f58-c3db69e15b66precedence_heuristic cell_id$61713e1c-8e37-45d9-9f58-c3db69e15b66downstream_cells_mapupstream_cells_mapchallenge_statement$3e1f7d5e-d294-11ef-2878-05744036f32cprecedence_heuristic cell_id$3e1f7d5e-d294-11ef-2878-05744036f32cdownstream_cells_mapupstream_cells_map@md_strgetindex$3e1b7d14-d294-11ef-0d10-1148a928dd57precedence_heuristic cell_id$3e1b7d14-d294-11ef-0d10-1148a928dd57downstream_cells_mapupstream_cells_map@md_strgetindex$2292c667-862e-4924-8f15-de0a8893c439precedence_heuristic cell_id$2292c667-862e-4924-8f15-de0a8893c439downstream_cells_mapupstream_cells_maphide_solution@md_strgetindex$3e1ed1a4-d294-11ef-2de4-d7cc540e06a1precedence_heuristic cell_id$3e1ed1a4-d294-11ef-2de4-d7cc540e06a1downstream_cells_mapupstream_cells_map@md_strgetindex$dd31ec7c-708d-4fd7-958d-f9887798a5bcprecedence_heuristic cell_id$dd31ec7c-708d-4fd7-958d-f9887798a5bcdownstream_cells_mapupstream_cells_map@md_strgetindex$577c6e08-9b74-43a4-9e27-9c477980406bprecedence_heuristic cell_id$577c6e08-9b74-43a4-9e27-9c477980406bdownstream_cells_mapNormalBind$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52cupstream_cells_mapHypertextLiteralHypertextLiteral.content@htl&PlutoUI.Experimental.transformed_valuePlutoUI.combineNormalStringHypertextLiteral.ResultPlutoUIHypertextLiteral.Bypass HypertextLiteral.attribute_value$7d493e09-f7cc-4e13-a506-b792edcbf390precedence_heuristic cell_id$7d493e09-f7cc-4e13-a506-b792edcbf390downstream_cells_mapupstream_cells_map@md_strgetindex$727dc817-0284-4c0f-9a92-21dcbea50807precedence_heuristic cell_id$727dc817-0284-4c0f-9a92-21dcbea50807downstream_cells_mapupstream_cells_maphide_solution@md_strgetindex$3e1e2b96-d294-11ef-3a68-fdc78232142eprecedence_heuristic cell_id$3e1e2b96-d294-11ef-3a68-fdc78232142edownstream_cells_mapupstream_cells_map@md_strgetindex$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4precedence_heuristic cell_id$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4downstream_cells_mapupstream_cells_maphide_solution@md_strgetindex$3e18b2fa-d294-11ef-1255-df048f0dcec2precedence_heuristic cell_id$3e18b2fa-d294-11ef-1255-df048f0dcec2downstream_cells_mapupstream_cells_map@md_strgetindex$3e185ab0-d294-11ef-3f7d-9bd465518274precedence_heuristic cell_id$3e185ab0-d294-11ef-3f7d-9bd465518274downstream_cells_mapupstream_cells_mapprevalence_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5asensitivity_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5a@md_strspecificity_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5agetindex$3e1c0e80-d294-11ef-0d19-375e01988f16precedence_heuristic cell_id$3e1c0e80-d294-11ef-0d19-375e01988f16downstream_cells_mapupstream_cells_map@md_strgetindex$5da42e9a-4318-48ea-9f43-4c1e1c97bcebprecedence_heuristic cell_id$5da42e9a-4318-48ea-9f43-4c1e1c97bcebdownstream_cells_mapupstream_cells_mapkeyconceptsummary$a8d4a517-84a7-426e-a49e-482c5fd047aeprecedence_heuristic cell_id$a8d4a517-84a7-426e-a49e-482c5fd047aedownstream_cells_mapupstream_cells_map@md_strgetindex$3e196d6a-d294-11ef-0795-41c045079251precedence_heuristic cell_id$3e196d6a-d294-11ef-0795-41c045079251downstream_cells_mapupstream_cells_map@md_strgetindex$218d3b6e-50b6-4b98-a00c-a19dd33d2c03precedence_heuristic cell_id$218d3b6e-50b6-4b98-a00c-a19dd33d2c03downstream_cells_mapupstream_cells_map@md_strgetindex$3e1b4b1c-d294-11ef-0423-9152887cc403precedence_heuristic cell_id$3e1b4b1c-d294-11ef-0423-9152887cc403downstream_cells_mapupstream_cells_map@md_strgetindex$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52cprecedence_heuristic cell_id$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52cdownstream_cells_mapy$34f1f3cb-9346-4bd3-ae9f-a25b4d5de781$c0ea3253-a06b-426c-91a3-a6dd33e42779x$34f1f3cb-9346-4bd3-ae9f-a25b4d5de781$c0ea3253-a06b-426c-91a3-a6dd33e42779upstream_cells_mapHypertextLiteralHypertextLiteral.content@htlPlutoRunnerPlutoRunner.Core.applicableScrubbable@bindHypertextLiteral.Result#PlutoRunner.PlutoRunner.create_bondHypertextLiteral.BypassNormalBind$577c6e08-9b74-43a4-9e27-9c477980406bPlutoRunner.Base.get$3e1eeb14-d294-11ef-1702-f5d2cf6fe60aprecedence_heuristic cell_id$3e1eeb14-d294-11ef-1702-f5d2cf6fe60adownstream_cells_mapupstream_cells_map@md_strgetindex$922770f4-ddc8-4089-b378-f14088276b43precedence_heuristic cell_id$922770f4-ddc8-4089-b378-f14088276b43downstream_cells_mapupstream_cells_mapexercise_statement$f8c8ba53-df36-48a6-afde-2952cbcfbe48precedence_heuristic cell_id$f8c8ba53-df36-48a6-afde-2952cbcfbe48downstream_cells_mapupstream_cells_map@md_strgetindex$41bee964-a0a9-4a7f-8505-54a9ee12ef0dprecedence_heuristic cell_id$41bee964-a0a9-4a7f-8505-54a9ee12ef0ddownstream_cells_mapupstream_cells_map@md_strgetindex$3e1b8bf4-d294-11ef-04cc-6364e46fdd64precedence_heuristic cell_id$3e1b8bf4-d294-11ef-04cc-6364e46fdd64downstream_cells_mapupstream_cells_map@md_strgetindex$98fa17a6-7c8b-46e4-b32d-52db183d88f8precedence_heuristic cell_id$98fa17a6-7c8b-46e4-b32d-52db183d88f8downstream_cells_mapupstream_cells_map@md_strgetindex$3e1f4f46-d294-11ef-29b8-69e546763781precedence_heuristic cell_id$3e1f4f46-d294-11ef-29b8-69e546763781downstream_cells_mapupstream_cells_map@md_strgetindex$3a1d380e-df80-4727-9772-f199214cf05dprecedence_heuristic cell_id$3a1d380e-df80-4727-9772-f199214cf05ddownstream_cells_mapupstream_cells_map@md_strgetindex$3e1e5a5a-d294-11ef-2fdf-efee4eb1a0f2precedence_heuristic cell_id$3e1e5a5a-d294-11ef-2fdf-efee4eb1a0f2downstream_cells_mapupstream_cells_map@md_strgetindex$3e1bdd02-d294-11ef-19e8-2f44eccf58afprecedence_heuristic cell_id$3e1bdd02-d294-11ef-19e8-2f44eccf58afdownstream_cells_mapupstream_cells_map@md_strgetindex$fea8ae4c-8ef9-4b74-ad13-1314afef97deprecedence_heuristic cell_id$fea8ae4c-8ef9-4b74-ad13-1314afef97dedownstream_cells_mapupstream_cells_map@md_strgetindex$db73766d-643c-41d7-a1eb-f376c657f860precedence_heuristic cell_id$db73766d-643c-41d7-a1eb-f376c657f860downstream_cells_mapupstream_cells_map@md_strgetindex$3b1b0869-b815-4697-9dba-3c4b4cb5ac47precedence_heuristic cell_id$3b1b0869-b815-4697-9dba-3c4b4cb5ac47downstream_cells_mapupstream_cells_maphide_solution@md_strgetindex$a8046381-ff11-40af-ae2b-078d71c586e7precedence_heuristic cell_id$a8046381-ff11-40af-ae2b-078d71c586e7downstream_cells_mapresult$2156f96e-eebe-4190-8ce9-c76825c6da71upstream_cells_map/specificity$4f6dd225-c64d-4b76-b075-0bf71c863b5a+sensitivity$4f6dd225-c64d-4b76-b075-0bf71c863b5a-prevalence$4f6dd225-c64d-4b76-b075-0bf71c863b5a*$3e1d20e0-d294-11ef-2044-e1fe6590a600precedence_heuristic cell_id$3e1d20e0-d294-11ef-2044-e1fe6590a600downstream_cells_mapupstream_cells_map@md_strgetindex$fc733d61-fd0f-4a13-9afc-4505ac0253dfprecedence_heuristic cell_id$fc733d61-fd0f-4a13-9afc-4505ac0253dfdownstream_cells_mapθ_bond$d93f73d4-2783-4777-b0ce-cdc0444cb300$b7445b9b-7fbb-4560-b947-a23af0fcf101f$8a7dd8b7-5faf-4091-8451-9769f842accbθ$079157c9-5d97-4dbc-8c47-afa8b661db06$8a7dd8b7-5faf-4091-8451-9769f842accbupstream_cells_map^PlutoRunner-Core.applicableCoreScrubbableBase.get@bindPlutoRunner.create_bond:Base*$3e1c60ba-d294-11ef-3a01-cf9e97512857precedence_heuristic cell_id$3e1c60ba-d294-11ef-3a01-cf9e97512857downstream_cells_mapupstream_cells_map@md_strgetindex$3e1bf116-d294-11ef-148b-f7a1ca3f3badprecedence_heuristic cell_id$3e1bf116-d294-11ef-148b-f7a1ca3f3baddownstream_cells_mapupstream_cells_map@md_strgetindex$3e1fa04a-d294-11ef-00c3-a51d1aaa5553precedence_heuristic cell_id$3e1fa04a-d294-11ef-00c3-a51d1aaa5553downstream_cells_mapupstream_cells_map@md_strgetindex$9b85a92b-c56b-48a3-97c1-6b1882e33a22precedence_heuristic cell_id$9b85a92b-c56b-48a3-97c1-6b1882e33a22downstream_cells_mapshow_disease_diagnosis_solution$2156f96e-eebe-4190-8ce9-c76825c6da71upstream_cells_mapHypertextLiteralHypertextLiteral.content@htlCheckBoxPlutoRunnerPlutoRunner.Core.applicable@bindHypertextLiteral.Result#PlutoRunner.PlutoRunner.create_bondHypertextLiteral.BypassPlutoRunner.Base.get$079157c9-5d97-4dbc-8c47-afa8b661db06precedence_heuristic cell_id$079157c9-5d97-4dbc-8c47-afa8b661db06downstream_cells_mapupstream_cells_mapHypertextLiteralHypertextLiteral.contentMarkdownLiteraln$4a81342c-17c7-4eb9-933b-edb98df7b9c4HypertextLiteral.Result-@mdx$42b47af6-b850-4987-a2d7-805a2cb64e43 MarkdownLiteral.CMParsedRendererθ$fc733d61-fd0f-4a13-9afc-4505ac0253dfHypertextLiteral.Bypass$3e1c70be-d294-11ef-14ed-0d46515541c5precedence_heuristic cell_id$3e1c70be-d294-11ef-14ed-0d46515541c5downstream_cells_mapupstream_cells_map@md_strgetindex$3e1bffec-d294-11ef-2a49-9ff0f6331addprecedence_heuristic cell_id$3e1bffec-d294-11ef-2a49-9ff0f6331adddownstream_cells_mapupstream_cells_map@md_strgetindex$3e198ba6-d294-11ef-3fe7-d70bf4833fa6precedence_heuristic cell_id$3e198ba6-d294-11ef-3fe7-d70bf4833fa6downstream_cells_mapupstream_cells_map@md_strgetindex$3e1a8eca-d294-11ef-1ef0-c15b24d05990precedence_heuristic cell_id$3e1a8eca-d294-11ef-1ef0-c15b24d05990downstream_cells_mapupstream_cells_map@md_strgetindex$3e1f68fa-d294-11ef-31b2-e7670da8c08cprecedence_heuristic cell_id$3e1f68fa-d294-11ef-31b2-e7670da8c08cdownstream_cells_mapupstream_cells_map@md_strgetindex$d0f672bd-74c9-4be1-b5ea-8d4040edb5dfprecedence_heuristic cell_id$d0f672bd-74c9-4be1-b5ea-8d4040edb5dfdownstream_cells_mapupstream_cells_mapnavigate_prev_next$3e1c51e2-d294-11ef-2c6d-d32a98308c6fprecedence_heuristic cell_id$3e1c51e2-d294-11ef-2c6d-d32a98308c6fdownstream_cells_mapupstream_cells_map@md_strgetindex$3e1a7c8e-d294-11ef-1f97-55e608d49141precedence_heuristic cell_id$3e1a7c8e-d294-11ef-1f97-55e608d49141downstream_cells_mapupstream_cells_map@md_strgetindex$b305a905-06c2-4a15-8042-72ef6375720fprecedence_heuristiccell_id$b305a905-06c2-4a15-8042-72ef6375720fdownstream_cells_mapBmlipTeachingToolsupstream_cells_map$3e1d6d00-d294-11ef-1081-e11b8397eb91precedence_heuristic cell_id$3e1d6d00-d294-11ef-1081-e11b8397eb91downstream_cells_mapupstream_cells_mapprevalence_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5asensitivity_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5a@md_strspecificity_bond$4f6dd225-c64d-4b76-b075-0bf71c863b5agetindex$3e191b6c-d294-11ef-3174-d1b4b36e252bprecedence_heuristic cell_id$3e191b6c-d294-11ef-3174-d1b4b36e252bdownstream_cells_mapupstream_cells_map@md_strgetindex$4a81342c-17c7-4eb9-933b-edb98df7b9c4precedence_heuristic cell_id$4a81342c-17c7-4eb9-933b-edb98df7b9c4downstream_cells_mapn$079157c9-5d97-4dbc-8c47-afa8b661db06$2156f96e-eebe-4190-8ce9-c76825c6da71upstream_cells_map@sprintf$9f4125a2-d5d2-4acf-8bad-82f94af230e8precedence_heuristic cell_id$9f4125a2-d5d2-4acf-8bad-82f94af230e8downstream_cells_mapupstream_cells_map@md_strgetindexblockquote$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafprecedence_heuristic cell_id$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafdownstream_cells_mapupstream_cells_map@md_strgetindex$3e027ede-4ac1-4521-a026-dc00bfca4adfprecedence_heuristic cell_id$3e027ede-4ac1-4521-a026-dc00bfca4adfdownstream_cells_mapupstream_cells_mapexercise_statement$3e1889b8-d294-11ef-17bb-496655fbd618precedence_heuristic cell_id$3e1889b8-d294-11ef-17bb-496655fbd618downstream_cells_mapupstream_cells_map@md_strgetindex$6d07be25-53d0-46b9-b197-a3680d830952precedence_heuristic cell_id$6d07be25-53d0-46b9-b197-a3680d830952downstream_cells_mapupstream_cells_maphide_solution@md_strgetindex$5f377237-d9a5-4778-aa4d-1c6ce109b705precedence_heuristic cell_id$5f377237-d9a5-4778-aa4d-1c6ce109b705downstream_cells_mapupstream_cells_map@md_strgetindex$d3b003c6-70ca-419f-a343-e35b266323f3precedence_heuristic cell_id$d3b003c6-70ca-419f-a343-e35b266323f3downstream_cells_mapupstream_cells_maphide_solution@md_strgetindex$3e1fb370-d294-11ef-1fb6-63a41a024691precedence_heuristic cell_id$3e1fb370-d294-11ef-1fb6-63a41a024691downstream_cells_mapupstream_cells_map@md_strexercise_statementgetindexlast_save_timeAڹ& notebook_id$196c2d02-d465-11f1-b0e4-2dda9d798eaein_temp_dir¨metadatafrontmatterdescription[Review of probability theory as a foundation for rational reasoning and Bayesian inference.authornameBMLIPurlhttps://github.com/bmlipbondscell_execution_order $eeb9a1f5-b857-4843-920b-2e4a9656f66b$5394e37c-ae00-4042-8ada-3bbf32fbca9e$b305a905-06c2-4a15-8042-72ef6375720f$42b47af6-b850-4987-a2d7-805a2cb64e43$a66ab9df-897c-42e5-8b0f-c520ceaffa23$3e17df5e-d294-11ef-38c7-f573724871d8$bcb4be20-0439-4809-a166-8c50b6b9206b$3e1803d0-d294-11ef-0304-df2b9b698cd1$9b9be452-9681-43e8-bb09-cc8728df384f$9f4125a2-d5d2-4acf-8bad-82f94af230e8$f8c8ba53-df36-48a6-afde-2952cbcfbe48$61713e1c-8e37-45d9-9f58-c3db69e15b66$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcaf$41bee964-a0a9-4a7f-8505-54a9ee12ef0d$3e1889b8-d294-11ef-17bb-496655fbd618$3e18b2fa-d294-11ef-1255-df048f0dcec2$3e18c25c-d294-11ef-11bc-a93c2572b107$3e18d2ea-d294-11ef-35e9-2332dd31dbf0$dd11e93a-3dad-4e97-8642-fb70edfa6aae$3e18e4bc-d294-11ef-38bc-cb97cb4e0963$3e18f18c-d294-11ef-33e4-b7f9495e0508$3e1906ea-d294-11ef-236e-c966a9474170$3e191b6c-d294-11ef-3174-d1b4b36e252b$3e192ef4-d294-11ef-1fc4-87175eeec5eb$3e19436c-d294-11ef-11c5-f9914f7a3a57$4edf38ab-a940-4ab0-be22-fa95cf571146$3e194ef2-d294-11ef-3b38-1ddc3063ff35$3e1964b4-d294-11ef-373d-712257fc130f$3e196d6a-d294-11ef-0795-41c045079251$3e198336-d294-11ef-26fd-03cd15876486$3e198ba6-d294-11ef-3fe7-d70bf4833fa6$3e19e95a-d294-11ef-3da4-6d23922a5150$3e1a69f4-d294-11ef-103e-efc47025fb8f$3e1a7c8e-d294-11ef-1f97-55e608d49141$3e1a8eca-d294-11ef-1ef0-c15b24d05990$3e1ab104-d294-11ef-1a98-412946949fba$fea8ae4c-8ef9-4b74-ad13-1314afef97de$3e1b4b1c-d294-11ef-0423-9152887cc403$3e1b5c9c-d294-11ef-137f-d75b3731eae4$3e1b7d14-d294-11ef-0d10-1148a928dd57$5377c5a4-77c4-4fa7-9f84-0c511e3bf708$3e1b8bf4-d294-11ef-04cc-6364e46fdd64$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0$3e1babca-d294-11ef-37c1-cd821a6488b2$3e1bba8e-d294-11ef-1f61-295af16078ce$3e1bcb00-d294-11ef-2795-bd225bd00496$3e1bdd02-d294-11ef-19e8-2f44eccf58af$3e1bf116-d294-11ef-148b-f7a1ca3f3bad$16c2eb59-16b8-4347-9aab-6e4b99016c79$3e1bffec-d294-11ef-2a49-9ff0f6331add$3e1c0e80-d294-11ef-0d19-375e01988f16$3e1c1e3e-d294-11ef-0955-bdf9d0ba3c53$3e1c4224-d294-11ef-2707-49470aaae6eb$3e1c51e2-d294-11ef-2c6d-d32a98308c6f$3e1c60ba-d294-11ef-3a01-cf9e97512857$3e1c70be-d294-11ef-14ed-0d46515541c5$3e1c806a-d294-11ef-1fad-17e5625279f7$3e1c9184-d294-11ef-3e35-5393d97fbc44$3e1d33c8-d294-11ef-0a08-bdc419949925$3e1b05ee-d294-11ef-33de-efed64d01c0d$b176ceae-884e-4460-9f66-020c1ac447f1$e3157dc0-5a64-4479-a37a-40fe25cccc07$7d493e09-f7cc-4e13-a506-b792edcbf390$3e1d20e0-d294-11ef-2044-e1fe6590a600$fc733d61-fd0f-4a13-9afc-4505ac0253df$d93f73d4-2783-4777-b0ce-cdc0444cb300$8a7dd8b7-5faf-4091-8451-9769f842accb$b7445b9b-7fbb-4560-b947-a23af0fcf101$922770f4-ddc8-4089-b378-f14088276b43$3e1de32c-d294-11ef-1f63-f190c8361404$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4$ff9142ba-3a85-48cf-8b78-07e0b554e280$3e027ede-4ac1-4521-a026-dc00bfca4adf$3e1e2b96-d294-11ef-3a68-fdc78232142e$727dc817-0284-4c0f-9a92-21dcbea50807$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4$ea803646-b18e-4f13-ab6a-fc5080d44e92$9b85a92b-c56b-48a3-97c1-6b1882e33a22$ef264651-854e-4374-8ea8-5476c85150c4$3e1e4dda-d294-11ef-33b7-4bbe3300ca22$3e1e5a5a-d294-11ef-2fdf-efee4eb1a0f2$3e1e7742-d294-11ef-1204-f9be24da07ab$6fafc778-c2c8-4e23-8d2e-6dc02e49e935$2292c667-862e-4924-8f15-de0a8893c439$3e1e9224-d294-11ef-38b3-137c2be22400$d2202628-e4f9-4289-b48e-23b5a0073f94$726e7e9f-3d68-4d02-9954-1edd98b74fdc$58f70d3e-4b64-414e-b560-327be2a0c4c2$3e1ea442-d294-11ef-1364-8dd9986325f7$6d07be25-53d0-46b9-b197-a3680d830952$3e1eba72-d294-11ef-2f53-b56f1862fcbb$3e1ed1a4-d294-11ef-2de4-d7cc540e06a1$3e1eeb14-d294-11ef-1702-f5d2cf6fe60a$3e1f225a-d294-11ef-04c6-f3ca018ab286$98fa17a6-7c8b-46e4-b32d-52db183d88f8$1cb8b2c4-e1ae-4973-ba53-fc6c7fe1f37a$218d3b6e-50b6-4b98-a00c-a19dd33d2c03$3e1f4f46-d294-11ef-29b8-69e546763781$3e1f68fa-d294-11ef-31b2-e7670da8c08c$3e1f7d5e-d294-11ef-2878-05744036f32c$3e1f8e48-d294-11ef-0f8a-b58294a8543d$3e1fa04a-d294-11ef-00c3-a51d1aaa5553$50bdc2fe-f48d-4c4e-8b4e-170782681366$db73766d-643c-41d7-a1eb-f376c657f860$3e1fb370-d294-11ef-1fb6-63a41a024691$317707a3-9ef1-4c67-b451-6adcfcff50f0$9b92fc89-2036-4525-979b-d296ab29329c$5da42e9a-4318-48ea-9f43-4c1e1c97bceb$03692f4d-0daf-4dfc-a7ff-6b954326e4d0$3a1d380e-df80-4727-9772-f199214cf05d$99d9099f-4908-4bb3-8d59-da9cb69af04c$3b1b0869-b815-4697-9dba-3c4b4cb5ac47$5f377237-d9a5-4778-aa4d-1c6ce109b705$5613e9b7-ff0d-435a-9de6-aaf293ebf592$fc3151f9-e143-4e31-b7b7-3f25b4fe9dab$66ebe33c-8360-4938-9b51-625e5bed176c$5b681e41-ad14-4c58-8ea0-4b6d85885c51$91dd40f0-c373-48b3-b83b-6e8df2c43e5a$a8d4a517-84a7-426e-a49e-482c5fd047ae$d3b003c6-70ca-419f-a343-e35b266323f3$d0f672bd-74c9-4be1-b5ea-8d4040edb5df$dd31ec7c-708d-4fd7-958d-f9887798a5bc$70d79732-0f55-40ba-929d-fba431318848$4f6dd225-c64d-4b76-b075-0bf71c863b5a$3e185ab0-d294-11ef-3f7d-9bd465518274$3e1d6d00-d294-11ef-1081-e11b8397eb91$a8046381-ff11-40af-ae2b-078d71c586e7$4a81342c-17c7-4eb9-933b-edb98df7b9c4$079157c9-5d97-4dbc-8c47-afa8b661db06$2156f96e-eebe-4190-8ce9-c76825c6da71$577c6e08-9b74-43a4-9e27-9c477980406b$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52c$34f1f3cb-9346-4bd3-ae9f-a25b4d5de781$c0ea3253-a06b-426c-91a3-a6dd33e42779process_statusreadycell_order $3e17df5e-d294-11ef-38c7-f573724871d8$bcb4be20-0439-4809-a166-8c50b6b9206b$3e1803d0-d294-11ef-0304-df2b9b698cd1$9b9be452-9681-43e8-bb09-cc8728df384f$9f4125a2-d5d2-4acf-8bad-82f94af230e8$f8c8ba53-df36-48a6-afde-2952cbcfbe48$61713e1c-8e37-45d9-9f58-c3db69e15b66$3e185ab0-d294-11ef-3f7d-9bd465518274$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcaf$41bee964-a0a9-4a7f-8505-54a9ee12ef0d$3e1889b8-d294-11ef-17bb-496655fbd618$3e18b2fa-d294-11ef-1255-df048f0dcec2$3e18c25c-d294-11ef-11bc-a93c2572b107$3e18d2ea-d294-11ef-35e9-2332dd31dbf0$dd11e93a-3dad-4e97-8642-fb70edfa6aae$3e18e4bc-d294-11ef-38bc-cb97cb4e0963$3e18f18c-d294-11ef-33e4-b7f9495e0508$3e1906ea-d294-11ef-236e-c966a9474170$3e191b6c-d294-11ef-3174-d1b4b36e252b$3e192ef4-d294-11ef-1fc4-87175eeec5eb$3e19436c-d294-11ef-11c5-f9914f7a3a57$4edf38ab-a940-4ab0-be22-fa95cf571146$3e194ef2-d294-11ef-3b38-1ddc3063ff35$3e1964b4-d294-11ef-373d-712257fc130f$3e196d6a-d294-11ef-0795-41c045079251$3e198336-d294-11ef-26fd-03cd15876486$3e198ba6-d294-11ef-3fe7-d70bf4833fa6$3e19e95a-d294-11ef-3da4-6d23922a5150$3e1a69f4-d294-11ef-103e-efc47025fb8f$3e1a7c8e-d294-11ef-1f97-55e608d49141$3e1a8eca-d294-11ef-1ef0-c15b24d05990$3e1ab104-d294-11ef-1a98-412946949fba$fea8ae4c-8ef9-4b74-ad13-1314afef97de$3e1b4b1c-d294-11ef-0423-9152887cc403$3e1b5c9c-d294-11ef-137f-d75b3731eae4$3e1b7d14-d294-11ef-0d10-1148a928dd57$5377c5a4-77c4-4fa7-9f84-0c511e3bf708$3e1b8bf4-d294-11ef-04cc-6364e46fdd64$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0$3e1babca-d294-11ef-37c1-cd821a6488b2$3e1bba8e-d294-11ef-1f61-295af16078ce$3e1bcb00-d294-11ef-2795-bd225bd00496$3e1bdd02-d294-11ef-19e8-2f44eccf58af$3e1bf116-d294-11ef-148b-f7a1ca3f3bad$16c2eb59-16b8-4347-9aab-6e4b99016c79$3e1bffec-d294-11ef-2a49-9ff0f6331add$3e1c0e80-d294-11ef-0d19-375e01988f16$3e1c1e3e-d294-11ef-0955-bdf9d0ba3c53$3e1c4224-d294-11ef-2707-49470aaae6eb$3e1c51e2-d294-11ef-2c6d-d32a98308c6f$3e1c60ba-d294-11ef-3a01-cf9e97512857$3e1c70be-d294-11ef-14ed-0d46515541c5$3e1c806a-d294-11ef-1fad-17e5625279f7$3e1c9184-d294-11ef-3e35-5393d97fbc44$3e1d33c8-d294-11ef-0a08-bdc419949925$3e1b05ee-d294-11ef-33de-efed64d01c0d$b176ceae-884e-4460-9f66-020c1ac447f1$e3157dc0-5a64-4479-a37a-40fe25cccc07$d93f73d4-2783-4777-b0ce-cdc0444cb300$079157c9-5d97-4dbc-8c47-afa8b661db06$7d493e09-f7cc-4e13-a506-b792edcbf390$8a7dd8b7-5faf-4091-8451-9769f842accb$b7445b9b-7fbb-4560-b947-a23af0fcf101$3e1d20e0-d294-11ef-2044-e1fe6590a600$fc733d61-fd0f-4a13-9afc-4505ac0253df$922770f4-ddc8-4089-b378-f14088276b43$3e1de32c-d294-11ef-1f63-f190c8361404$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4$ff9142ba-3a85-48cf-8b78-07e0b554e280$3e027ede-4ac1-4521-a026-dc00bfca4adf$3e1e2b96-d294-11ef-3a68-fdc78232142e$727dc817-0284-4c0f-9a92-21dcbea50807$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4$ea803646-b18e-4f13-ab6a-fc5080d44e92$3e1d6d00-d294-11ef-1081-e11b8397eb91$9b85a92b-c56b-48a3-97c1-6b1882e33a22$2156f96e-eebe-4190-8ce9-c76825c6da71$ef264651-854e-4374-8ea8-5476c85150c4$3e1e4dda-d294-11ef-33b7-4bbe3300ca22$3e1e5a5a-d294-11ef-2fdf-efee4eb1a0f2$3e1e7742-d294-11ef-1204-f9be24da07ab$6fafc778-c2c8-4e23-8d2e-6dc02e49e935$2292c667-862e-4924-8f15-de0a8893c439$3e1e9224-d294-11ef-38b3-137c2be22400$d2202628-e4f9-4289-b48e-23b5a0073f94$726e7e9f-3d68-4d02-9954-1edd98b74fdc$58f70d3e-4b64-414e-b560-327be2a0c4c2$3e1ea442-d294-11ef-1364-8dd9986325f7$6d07be25-53d0-46b9-b197-a3680d830952$3e1eba72-d294-11ef-2f53-b56f1862fcbb$3e1ed1a4-d294-11ef-2de4-d7cc540e06a1$3e1eeb14-d294-11ef-1702-f5d2cf6fe60a$3e1f225a-d294-11ef-04c6-f3ca018ab286$98fa17a6-7c8b-46e4-b32d-52db183d88f8$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52c$1cb8b2c4-e1ae-4973-ba53-fc6c7fe1f37a$34f1f3cb-9346-4bd3-ae9f-a25b4d5de781$218d3b6e-50b6-4b98-a00c-a19dd33d2c03$c0ea3253-a06b-426c-91a3-a6dd33e42779$3e1f4f46-d294-11ef-29b8-69e546763781$3e1f68fa-d294-11ef-31b2-e7670da8c08c$3e1f7d5e-d294-11ef-2878-05744036f32c$3e1f8e48-d294-11ef-0f8a-b58294a8543d$3e1fa04a-d294-11ef-00c3-a51d1aaa5553$50bdc2fe-f48d-4c4e-8b4e-170782681366$db73766d-643c-41d7-a1eb-f376c657f860$3e1fb370-d294-11ef-1fb6-63a41a024691$317707a3-9ef1-4c67-b451-6adcfcff50f0$9b92fc89-2036-4525-979b-d296ab29329c$5da42e9a-4318-48ea-9f43-4c1e1c97bceb$03692f4d-0daf-4dfc-a7ff-6b954326e4d0$3a1d380e-df80-4727-9772-f199214cf05d$99d9099f-4908-4bb3-8d59-da9cb69af04c$3b1b0869-b815-4697-9dba-3c4b4cb5ac47$5f377237-d9a5-4778-aa4d-1c6ce109b705$5613e9b7-ff0d-435a-9de6-aaf293ebf592$fc3151f9-e143-4e31-b7b7-3f25b4fe9dab$66ebe33c-8360-4938-9b51-625e5bed176c$5b681e41-ad14-4c58-8ea0-4b6d85885c51$91dd40f0-c373-48b3-b83b-6e8df2c43e5a$a8d4a517-84a7-426e-a49e-482c5fd047ae$d3b003c6-70ca-419f-a343-e35b266323f3$d0f672bd-74c9-4be1-b5ea-8d4040edb5df$dd31ec7c-708d-4fd7-958d-f9887798a5bc$eeb9a1f5-b857-4843-920b-2e4a9656f66b$5394e37c-ae00-4042-8ada-3bbf32fbca9e$b305a905-06c2-4a15-8042-72ef6375720f$70d79732-0f55-40ba-929d-fba431318848$4f6dd225-c64d-4b76-b075-0bf71c863b5a$a8046381-ff11-40af-ae2b-078d71c586e7$42b47af6-b850-4987-a2d7-805a2cb64e43$a66ab9df-897c-42e5-8b0f-c520ceaffa23$4a81342c-17c7-4eb9-933b-edb98df7b9c4$577c6e08-9b74-43a4-9e27-9c477980406bcell_inputs $3e1bba8e-d294-11ef-1f61-295af16078cemetadatadisabled©show_logsîskip_as_script§cell_id$3e1bba8e-d294-11ef-1f61-295af16078cecode_foldedäcodeumd"""
Of course, in the continuous domain, marginalization becomes
```math
p(X)=\int_Y p(X,Y) \,\mathrm{d}Y
```
"""$3e1eba72-d294-11ef-2f53-b56f1862fcbbmetadatadisabled©show_logsîskip_as_script§cell_id$3e1eba72-d294-11ef-2f53-b56f1862fcbbcode_foldedäcode}md"""
Clearly, it follows that if ``X`` and ``Y`` are **independent**, then
```math
\Sigma_z = \Sigma_x + \Sigma_y
```
"""$9b9be452-9681-43e8-bb09-cc8728df384fmetadatadisabled©show_logsîskip_as_script§cell_id$9b9be452-9681-43e8-bb09-cc8728df384fcode_foldedäcodemd"""
## 📕 Data Analysis: A Bayesian Tutorial
The following is an excerpt from the book [Data Analysis: A Bayesian Tutorial](https://global.oup.com/academic/product/data-analysis-9780198568322) (2006), by D.S. Sivia with J.S. Skilling:
"""$3e1f8e48-d294-11ef-0f8a-b58294a8543dmetadatadisabled©show_logsîskip_as_script§cell_id$3e1f8e48-d294-11ef-0f8a-b58294a8543dcode_foldedäcodezhide_proof(
md"""
```math
P_y(\hat{y}) = P(y=\hat{y}) = P(h(x)=\hat{y}) = P(x=g(\hat{y})) = P_x(g(\hat{y})) \,.
```
""")$5b681e41-ad14-4c58-8ea0-4b6d85885c51metadatadisabled©show_logsîskip_as_script§cell_id$5b681e41-ad14-4c58-8ea0-4b6d85885c51code_foldedäcodeGmd"""
##### Who Speaks the Truth? (***)
The inhabitants of an island tell the truth one-third of the time. They lie with probability ``2/3``. On an occasion, after one of them made a statement, you ask another person "was that statement true?" and he says "yes". What is the probability that the statement was indeed true?
"""$42b47af6-b850-4987-a2d7-805a2cb64e43metadatadisabled©show_logsîskip_as_script§cell_id$42b47af6-b850-4987-a2d7-805a2cb64e43code_folded¤code# The Disease Diagnosis example uses a combination of:
# - PlutoUI.Scrubbable for the interactive input percentages
# - MarkdownLiteral to be able to interpolate numbers into markdown math
# - PrintF for consistent formatting
using MarkdownLiteral: @mdx$726e7e9f-3d68-4d02-9954-1edd98b74fdcmetadatadisabled©show_logsîskip_as_script§cell_id$726e7e9f-3d68-4d02-9954-1edd98b74fdccode_foldedäcodekeyconcept("",
md"""
No matter the specification of ``p(x)``, the mean and covariance matrix for ``z=Ax + b`` are given by
```math
\begin{align}
\mu_z &= A\mu_x + b \\
\Sigma_z &= A\,\Sigma_x\,A^T
\end{align}
```
""")$3e18e4bc-d294-11ef-38bc-cb97cb4e0963metadatadisabled©show_logsîskip_as_script§cell_id$3e18e4bc-d294-11ef-38bc-cb97cb4e0963code_foldedäcode keyconcept(" ",
md"""
When assigning real numbers to **degrees of belief**, logical consistency forces adherence to the sum and product rules of probability theory. This makes probability theory the optimal framework for information processing in the face of uncertainty.
"""
)$e3157dc0-5a64-4479-a37a-40fe25cccc07metadatadisabled©show_logsîskip_as_script§cell_id$e3157dc0-5a64-4479-a37a-40fe25cccc07code_foldedäcodeOcode_example("Sampling Distribution and Likelihood Function for the Coin Toss")$3e1c806a-d294-11ef-1fad-17e5625279f7metadatadisabled©show_logsîskip_as_script§cell_id$3e1c806a-d294-11ef-1fad-17e5625279f7code_foldedäcode5md"""
In a machine learning context, often the data is observed, and ``\theta`` is the free variable. In that case, for given observations ``D=D_0``, the **likelihood function** (which is a function only of the model parameters ``\theta``) is defined as
```math
L(\theta) \triangleq p(D=D_0|\theta)
```
"""$3e1ea442-d294-11ef-1364-8dd9986325f7metadatadisabled©show_logsîskip_as_script§cell_id$3e1ea442-d294-11ef-1364-8dd9986325f7code_foldedäcode_md"""
You should now be able to derive the following: for any distribution of variables ``X`` and ``Y``, show that the mean and variance of the sum ``Z = X+Y`` is given by
```math
\begin{align*}
\mu_z &= \mu_x + \mu_y \\
\Sigma_z &= \Sigma_x + \Sigma_y + \Sigma_{xy} + \Sigma_{yx}
\end{align*}
```
where ``\Sigma_{yx} = \Sigma_{xy}^T``.
"""$3e1c4224-d294-11ef-2707-49470aaae6ebmetadatadisabled©show_logsîskip_as_script§cell_id$3e1c4224-d294-11ef-2707-49470aaae6ebcode_foldedäcodeMmd"""
For a given data set ``D``, the posterior probabilities of the parameters scale relatively against each other as
```math
p(\theta|D) \propto p(D|\theta) p(\theta)
```
Hence, all that we can learn from the observed data is contained in the likelihood function ``p(D|\theta)``. This is called the **likelihood principle**.
"""$3e1f225a-d294-11ef-04c6-f3ca018ab286metadatadisabled©show_logsîskip_as_script§cell_id$3e1f225a-d294-11ef-04c6-f3ca018ab286code_foldedäcodemd"""
$(code_example("Sum of Two Gaussian-distributed Variables"; big=true, header_level=2))
Consider two independent Gaussian-distributed variables ``X`` and ``Y`` (see [wikipedia normal distribution](https://en.wikipedia.org/wiki/Normal_distribution) for definition of a Gaussian (=Normal) distribution):
```math
\begin{align*}
p_X(x) &= \mathcal{N}(\,x\,|\,\mu_X,\sigma_X^2\,) \\
p_Y(y) &= \mathcal{N}(\,y\,|\,\mu_Y,\sigma_Y^2\,)
\end{align*}
```
Let ``Z = X + Y``. Performing the convolution (nice exercise) yields a Gaussian PDF for ``Z``:
```math
p_Z(z) = \mathcal{N}(\,z\,|\,\mu_X+\mu_Y,\sigma_X^2+\sigma_Y^2\,).
```
We illustrate the distributions for ``X``, ``Y`` and ``Z`` using Julia:
"""$5613e9b7-ff0d-435a-9de6-aaf293ebf592metadatadisabled©show_logsîskip_as_script§cell_id$5613e9b7-ff0d-435a-9de6-aaf293ebf592code_foldedäcodehide_solution(
md"""
The following probabilities are given in the problem statement,
```math
\begin{align}
p(b_1) &= p(b_2) = 1/2 \\
p(a|b_1) &= 8/12, \quad p(a|b_2) = 10/12 \\
p(o|b_1) &= 4/12, \quad p(o|b_2) = 2/12
\end{align}
```
(a)
```math
p(a) = \sum_i p(a,b_i) = \sum_i p(a|b_i)p(b_i)=\frac{8}{12}\cdot\frac{1}{2} + \frac{10}{12}\cdot\frac{1}{2} = \frac{3}{4}
```
(b)
```math
p(b_1|a) = \frac{p(a,b_1)}{p(a)} = \frac{p(a|b_1)p(b_1)}{p(a)} = \frac{\frac{8}{12}\cdot\frac{1}{2}}{\frac{3}{4}} = \frac{4}{9}
```
"""
)$3e1c1e3e-d294-11ef-0955-bdf9d0ba3c53metadatadisabled©show_logsîskip_as_script§cell_id$3e1c1e3e-d294-11ef-0955-bdf9d0ba3c53code_foldedäcodemd"""
Hence, having access to likelihood and prior is in principle sufficient to compute both the evidence and the posterior. To emphasize that point, Bayes rule is sometimes written as a transformation:
```math
\underbrace{\underbrace{p(\theta|D)}_{\text{posterior}}\cdot \underbrace{p(D)}_{\text{evidence}}}_{\text{this is what we want to compute}} = \underbrace{\underbrace{p(D|\theta)}_{\text{likelihood}}\cdot \underbrace{p(\theta)}_{\text{prior}}}_{\text{this is available}}
```
"""$9b92fc89-2036-4525-979b-d296ab29329cmetadatadisabled©show_logsîskip_as_script§cell_id$9b92fc89-2036-4525-979b-d296ab29329ccode_foldedäcodemd"""
# Summary
"""$eeb9a1f5-b857-4843-920b-2e4a9656f66bmetadatadisabled©show_logsîskip_as_script§cell_id$eeb9a1f5-b857-4843-920b-2e4a9656f66bcode_folded¤codeusing Plots, LaTeXStrings$5377c5a4-77c4-4fa7-9f84-0c511e3bf708metadatadisabled©show_logsîskip_as_script§cell_id$5377c5a4-77c4-4fa7-9f84-0c511e3bf708code_foldedäcode'hide_proof(
md"""
```math
\begin{align*}
\sum_i p(A,B_i) &= p\big(\sum_i AB_i\big) &&\quad \text{(since all $AB_i$ are disjoint)}\\
&= p\big(A,\sum_i B_i\big) \\
&= p(A,\Omega) &&\quad \text{($\Omega$ is true event, since $B_i$ are exhaustive)} \\
&= p(A)
\end{align*}
```
""")$3e1e7742-d294-11ef-1204-f9be24da07abmetadatadisabled©show_logsîskip_as_script§cell_id$3e1e7742-d294-11ef-1204-f9be24da07abcode_foldedäcodemd"""
The **covariance** matrix between *vectors* ``x`` and ``y`` is a mixed central moment, defined as
```math
\begin{align*}
\Sigma_{xy} &\triangleq \mathbb{E}\left[ (x-\mu_x) (y-\mu_y)^T \right]\\
&= \mathbb{E}\left[ (x-\mu_x) (y^T-\mu_y^T) \right]\\
&= \mathbb{E}[x y^T] - \mu_x \mu_y^T
\end{align*}
```
Clearly, if ``x`` and ``y`` are independent, then ``\Sigma_{xy} = 0``, since in that case ``\mathbb{E}[x y^T] = \mathbb{E}[x] \mathbb{E}[y^T] = \mu_x \mu_y^T``.
"""$ff9142ba-3a85-48cf-8b78-07e0b554e280metadatadisabled©show_logsîskip_as_script§cell_id$ff9142ba-3a85-48cf-8b78-07e0b554e280code_foldedäcodeumd"""
Note that the physical state of the bag—either containing one black ball or one white ball—remains unchanged after the “put one in, take one out” procedure.
Yet, the probability we assign to the color of the ball in the bag changes.
This illustrates a key point: probabilities describe a person’s state of knowledge, not an intrinsic property of nature.
"""$ea803646-b18e-4f13-ab6a-fc5080d44e92metadatadisabled©show_logsîskip_as_script§cell_id$ea803646-b18e-4f13-ab6a-fc5080d44e92code_foldedäcode6challenge_solution("Disease Diagnosis",header_level=1)$3e192ef4-d294-11ef-1fc4-87175eeec5ebmetadatadisabled©show_logsîskip_as_script§cell_id$3e192ef4-d294-11ef-1fc4-87175eeec5ebcode_foldedäcodemd"""
## Frequentist vs. Bayesian Interpretation of Probabilities
The interpretation of a probability as a **degree-of-belief** about the truth value of an event is also called the **Bayesian** interpretation.
"""$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0metadatadisabled©show_logsîskip_as_script§cell_id$3e1b9ba8-d294-11ef-18f2-db8eed3d87d0code_foldedäcodeٹmd"""
Summing ``Y`` out of a joint distribution ``p(X,Y)`` is called **marginalization** and the result ``p(X)`` is sometimes referred to as the **marginal probability** of ``X``.
"""$317707a3-9ef1-4c67-b451-6adcfcff50f0metadatadisabled©show_logsîskip_as_script§cell_id$317707a3-9ef1-4c67-b451-6adcfcff50f0code_foldedäcode|hide_solution(
md"""
Note that ``h(x)`` is invertible with ``x = g(y) = \sigma y + \mu``. The change-of-variable formula leads to
```math
\begin{align*}
p_y(y) &= p_x(g(y)) \cdot g^\prime(y) \\
&= p_x(\sigma y + \mu) \cdot \sigma \\
&= \frac{1}{\sigma\sqrt{2 \pi}} \exp\left( - \frac{(\sigma y + \mu - \mu)^2}{2\sigma^2}\right) \cdot \sigma \\
&= \frac{1}{\sqrt{2 \pi}} \exp\left( - \frac{y^2 }{2}\right)\\
&= \mathcal{N}(y|0,1)
\end{align*}
```
In the statistics literature, ``y = \frac{x-\mu}{\sigma}`` is called the **standardized** variable since it transforms a general normal variable into a standard normal one.)
""")$8a7dd8b7-5faf-4091-8451-9769f842accbmetadatadisabled©show_logsîskip_as_script§cell_id$8a7dd8b7-5faf-4091-8451-9769f842accbcode_foldedäcodelet
p1 = plot(
[0,1], f([0,1], θ);
line=:stem,
marker=:circle,
xrange=(-0.5, 1.5), yrange=(0,1),
title="Sampling Distribution",
xlabel="y", ylabel=L"p(y|θ=%$θ)", label=""
)
_θ = 0:0.01:1
y=1
p2 = plot(
_θ, f(y, _θ);
ylabel=L"p(y=%$y | θ)", xlabel=L"θ",
title="Likelihood Function", label=""
)
scatter!(p2,
[θ], [f(y, θ)];
label=nothing,
)
plot(p1, p2)
end$99d9099f-4908-4bb3-8d59-da9cb69af04cmetadatadisabled©show_logsîskip_as_script§cell_id$99d9099f-4908-4bb3-8d59-da9cb69af04ccode_foldedäcodeuhint(
md"""
Here, you may make use of the (Boolean logic) fact that ``A + B = \overline {\bar A \bar B }``.
"""
)$dd11e93a-3dad-4e97-8642-fb70edfa6aaemetadatadisabled©show_logsîskip_as_script§cell_id$dd11e93a-3dad-4e97-8642-fb70edfa6aaecode_foldedäcodemd"""
##### Some notational conventions
In the above sum and product rules,
- the **conditional probability** of ``A`` given ``I``, denoted by ``p(A|I)``, indicates the degree of belief in event ``A``, given that ``I`` is true.
- ``p(A,B|I)`` should be read as the *joint* probabability that both ``A`` and ``B`` are true, given that ``I`` is true.
- Similarly, ``p(A|B,I)`` is the probability that ``A`` is true, given that both ``B`` and ``I`` are true.
"""$3e1de32c-d294-11ef-1f63-f190c8361404metadatadisabled©show_logsîskip_as_script§cell_id$3e1de32c-d294-11ef-1f63-f190c8361404code_foldedäcode md"""
##### Problem
- A bag contains one ball, known to be either white or black. A white ball is put in , and the bag is shaken. Next, a ball is drawn out, which proves to be white. If we now take out another ball, what is the probability it will be white?
"""$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4metadatadisabled©show_logsîskip_as_script§cell_id$fae6f2ce-ac8f-4ea6-b2cf-38b30a7e20d4code_foldedäcodemd"""
In this case, knowledge about the future influences our state of knowledge about the present. Once again, we see that conditional probabilities capture implications for a state of knowledge, not temporal causality.
"""$3e1a69f4-d294-11ef-103e-efc47025fb8fmetadatadisabled©show_logsîskip_as_script§cell_id$3e1a69f4-d294-11ef-103e-efc47025fb8fcode_foldedäcodemd"""
If ``X`` is a variable, then an *assignment* ``X=x`` (where ``x`` is a value, e.g., ``X=5``) can be interpreted as an event. Hence, the expression ``p(X=5)`` should be interpreted as the *degree-of-belief of the event* that variable ``X`` takes on the value ``5``.
"""$3e198336-d294-11ef-26fd-03cd15876486metadatadisabled©show_logsîskip_as_script§cell_id$3e198336-d294-11ef-26fd-03cd15876486code_foldedäcode(md"""
Aubrey Clayton, in his wonderful book [Bernoulli's fallacy](https://aubreyclayton.com/bernoulli.html) (2021), writes about this issue:
> “Compared with Bayesian methods, standard [frequentist] statistical techniques use only a small fraction of the available information about a research hypothesis (how well it predicts some observation), so naturally they will struggle when that limited information proves inadequate. Using standard statistical methods is like driving a car at night on a poorly lit highway: to keep from going in a ditch, we could build an elaborate system of bumpers and guardrails and equip the car with lane departure warnings and sophisticated navigation systems, and even then we could at best only drive to a few destinations. Or we could turn on the headlights.”
"""$6fafc778-c2c8-4e23-8d2e-6dc02e49e935metadatadisabled©show_logsîskip_as_script§cell_id$6fafc778-c2c8-4e23-8d2e-6dc02e49e935code_foldedäcode@md"""
Exercise: Prove that ``\Sigma_{xy} = \Sigma_{yx}^{T}``
"""$5394e37c-ae00-4042-8ada-3bbf32fbca9emetadatadisabled©show_logsîskip_as_script§cell_id$5394e37c-ae00-4042-8ada-3bbf32fbca9ecode_folded¤codeusing Distributions$3e1ab104-d294-11ef-1a98-412946949fbametadatadisabled©show_logsîskip_as_script§cell_id$3e1ab104-d294-11ef-1a98-412946949fbacode_foldedäcodeQmd"""
# $(HTML("
Probability Theory Calculus "))
"""$c0ea3253-a06b-426c-91a3-a6dd33e42779metadatadisabled©show_logsîskip_as_script§cell_id$c0ea3253-a06b-426c-91a3-a6dd33e42779code_foldedäcodelet
# Calculate the x-range for plotting
# range_min = min(μx-2*σx, μy-2*σy, μz-2*σz)
# range_max = max(μx+2*σx, μy+2*σy, μz+2*σz)
range_min = 0
range_max = 6
range_grid = range(range_min, stop=range_max, length=100)
plot(ylim=(0,1))
plot!(range_grid, t -> pdf(x,t), label=L"X", lw=2, fill=(0, 0.1))
plot!(range_grid, t -> pdf(y,t), label=L"Y", lw=2, fill=(0, 0.1))
plot!(range_grid, t -> pdf(z,t), label=L"Z", lw=2, fill=(0, 0.1), linestyle=:dash)
end$b176ceae-884e-4460-9f66-020c1ac447f1metadatadisabled©show_logsîskip_as_script§cell_id$b176ceae-884e-4460-9f66-020c1ac447f1code_foldedäcodemd"""
# Examples
"""$d2202628-e4f9-4289-b48e-23b5a0073f94metadatadisabled©show_logsîskip_as_script§cell_id$d2202628-e4f9-4289-b48e-23b5a0073f94code_foldedäcode_hide_proof(
md"""
Let ``\mathbb{E}[\cdot]`` refer to the expectation (mean) operator. By linearity of expectation and the fact that ``A`` and ``b`` are constants,
```math
\mu_z = \mathbb{E}[z] = \mathbb{E}[A x + b] = A\,\mathbb{E}[x] + b = A\mu_x + b\,.
```
For the covariance matrix,
```math
\begin{align}
\Sigma_z &= \mathbb{E}\big[(z-\mu_z)(z-\mu_z)^T \big] \\
&= \mathbb{E}\big[(Ax + b - (A\mu_x + b))(Ax + b - (A\mu_x + b))^T \big] \\
&= \mathbb{E}\big[(Ax - A\mu_x)(Ax - A\mu_x)^T \big] \\
&= A\,\mathbb{E}\big[(x-\mu_x)(x-\mu_x)^\top\big]A^T \\
&= A\Sigma_x A^T \,.
\end{align}
```
"""
)$b7445b9b-7fbb-4560-b947-a23af0fcf101metadatadisabled©show_logsîskip_as_script§cell_id$b7445b9b-7fbb-4560-b947-a23af0fcf101code_foldedäcodeDmd"""
Click and drag this number to change ``\theta``: $θ_bond.
"""$bcb4be20-0439-4809-a166-8c50b6b9206bmetadatadisabled©show_logsîskip_as_script§cell_id$bcb4be20-0439-4809-a166-8c50b6b9206bcode_foldedäcodePlutoUI.TableOfContents()$50bdc2fe-f48d-4c4e-8b4e-170782681366metadatadisabled©show_logsîskip_as_script§cell_id$50bdc2fe-f48d-4c4e-8b4e-170782681366code_foldedäcodePhide_proof(
md"""
We assume again that ``y = h(x)`` is a one-to-one function with ``x = g(y) = h^{-1}(y)``. Let ``a=g(c)`` and ``b=g(d)``. Then
```math
\begin{align}
P(a ≤ x ≤ b) &= \int_a^b p_x(x)\mathrm{d}x \\
&= \int_{g(c)}^{g(d)} p_x(x)\mathrm{d}x \\
&= \int_c^d p_x(g(y))\mathrm{d}g(y) \\
&= \int_c^d \underbrace{p_x(g(y)) g^\prime(y)}_{p_y(y)}\mathrm{d}y \\
&= P(c ≤ y ≤ d)
\end{align}
```
Equating the two probability masses, ``p_y(y)\mathrm{d}y = p_x(x)\mathrm{d}x``, leads to the identification of the relation
```math
p_y(y) = p_x(g(y)) g^\prime(y)\,,
```
"""
)$fc3151f9-e143-4e31-b7b7-3f25b4fe9dabmetadatadisabled©show_logsîskip_as_script§cell_id$fc3151f9-e143-4e31-b7b7-3f25b4fe9dabcode_foldedäcodeumd"""
##### What is a Random Signal? (*)
Is a speech signal a "probabilistic" (random) or a deterministic signal?
"""$1cb8b2c4-e1ae-4973-ba53-fc6c7fe1f37ametadatadisabled©show_logsîskip_as_script§cell_id$1cb8b2c4-e1ae-4973-ba53-fc6c7fe1f37acode_foldedäcodeGmd"""
Compute the parameters for the distribution of ``Z = X + Y``:
"""$d93f73d4-2783-4777-b0ce-cdc0444cb300metadatadisabled©show_logsîskip_as_script§cell_id$d93f73d4-2783-4777-b0ce-cdc0444cb300code_foldedäcodeJmd"""
Consider the following simple model for the outcome ``y \in \{0,1\}`` (tail = ``0``, head = ``1``) of a biased coin toss with a real parameter $θ_bond ``= \theta \in [0,1]``:
```math
\begin{align*}
p(y|\theta) = \theta^y (1-\theta)^{1-y}\\
\end{align*}
```
Next, we use Julia to plot both the sampling distribution
"""$3e1803d0-d294-11ef-0304-df2b9b698cd1metadatadisabled©show_logsîskip_as_script§cell_id$3e1803d0-d294-11ef-0304-df2b9b698cd1code_foldedäcodemd"""
## Preliminaries
##### Goal
- Review of Probability Theory as a theory for rational/logical reasoning with uncertainties (i.e., a Bayesian interpretation)
##### Materials
- Mandatory
- These lecture notes
- Optional
- Bishop pp. 12-24
- [3Blue1Brown, YouTube video on Bayes theorem (2019)](https://youtu.be/HZGCoVF3YvM?si=JaXdesPjU8B_BtrC)
- Nice animated tutorial on Bayes rule.
- [Edwin Jaynes, Probability Theory–The Logic of Science (2003)](https://bayes.wustl.edu/etj/science.pdf.html)
- Brilliant book on the Bayesian view of probability theory. Just for fun, scan the annotated bibliography and references.
- [Aubrey Clayton, Bernoulli's Fallacy–Statistical Illogic and the Crisis of Modern Science (2021)](https://aubreyclayton.com/bernoulli.html)
- A very readable account of the history of statistics and probability theory. Discusses why most popular statistics recipes are very poor scientific analysis tools. Use probability theory instead!
- [Ariel Caticha, Entropic Inference and the Foundations of Physics (2012)](https://github.com/bmlip/course/blob/main/assets/files/Caticha-2012-Entropic-Inference-and-the-Foundations-of-Physics.pdf), pp.7-56 (ch.2: probability)
- Great introduction to probability theory, in particular w.r.t. its correct interpretation as a state-of-knowledge.
- Absolutely worth your time to read the whole chapter, even if you skip section 2.2.4 (pp.15-18) on Cox's proof.
- [Joram Soch et al ., The Book of Statistical Proofs (2023 - )](https://statproofbook.github.io/)
- Online resource for proofs in probability theory and statistical inference.
"""$3e18d2ea-d294-11ef-35e9-2332dd31dbf0metadatadisabled©show_logsîskip_as_script§cell_id$3e18d2ea-d294-11ef-35e9-2332dd31dbf0code_foldedäcodemd"""
Under these assumptions, Cox showed that any consistent system of reasoning about uncertainty must obey the **rules of probability theory** (see [Cox theorem, 1946](https://en.wikipedia.org/wiki/Cox%27s_theorem), and [Caticha, 2012](https://github.com/bmlip/course/blob/main/assets/files/Caticha-2012-Entropic-Inference-and-the-Foundations-of-Physics.pdf), pp.7-26). These rules are the sum and product rules:
##### The sum rule
- The degree of belief in the disjunction of two events ``A`` and ``B``, with given background information ``I``, is evaluated as
```math
p(A+B|I) = p(A|I) + p(B|I) - p(A,B|I)
```
##### The product rule
- The degree of belief in the conjunction of two events ``A`` and ``B``, with given background information ``I``, is evaluated as
```math
p(A,B|I) = p(A|B,I)\,p(B|I)
```
Cox’s Theorem derives the rules of probability theory from first principles, not as arbitrary postulates but as consequences of rational reasoning.
In other words: **Probability = extended logic**.
"""$2156f96e-eebe-4190-8ce9-c76825c6da71metadatadisabled©show_logsîskip_as_script§cell_id$2156f96e-eebe-4190-8ce9-c76825c6da71code_foldedäcodeif show_disease_diagnosis_solution
@mdx("""
- The given information is ``p(D=1)=$(n(prevalence))``, ``p(T=1|D=1)=$(n(sensitivity))`` and ``p(T=0|D=0)=$(n(specificity))``. We are asked to derive ``p( D=1 | T=1)``. We just follow the sum and product rules to derive the requested probability:
```math
\\begin{align*}
p( D=1 &| T=1) \\\\
&\\stackrel{p}{=} \\frac{p(T=1,D=1)}{p(T=1)} \\\\
&\\stackrel{p}{=} \\frac{p(T=1|D=1)p(D=1)}{p(T=1)} \\\\
&\\stackrel{s}{=} \\frac{p(T=1|D=1)p(D=1)}{p(T=1|D=1)p(D=1)+p(T=1|D=0)p(D=0)} \\\\
&= \\frac{$(n(sensitivity))\\times$(n(prevalence))}{$(n(sensitivity))\\times$(n(prevalence)) + $(n(1 - specificity))\\times$(n(1 - prevalence))} = \\boldsymbol{$(n(result; digits=4))}
\\end{align*}
```
Note that ``p(\\text{sick}|\\text{positive test}) = $(n(result))`` while ``p(\\text{positive test} | \\text{sick}) = $(n(sensitivity))``. This is a huge difference that is sometimes called the "medical test paradox" or the [base rate fallacy](https://en.wikipedia.org/wiki/Base_rate_fallacy).
Many people have trouble distinguishing ``p(A|B)`` from ``p(B|A)`` in their heads. This has led to major negative consequences. For instance, unfounded convictions in the legal arena and numerous unfounded conclusions in the pursuit of scientific results. See [Ioannidis (2005)](https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124) and [Clayton (2021)](https://aubreyclayton.com/bernoulli.html).
""")
end$ef264651-854e-4374-8ea8-5476c85150c4metadatadisabled©show_logsîskip_as_script§cell_id$ef264651-854e-4374-8ea8-5476c85150c4code_foldedäcode!md"# Moments and Transformations"$91dd40f0-c373-48b3-b83b-6e8df2c43e5ametadatadisabled©show_logsîskip_as_script§cell_id$91dd40f0-c373-48b3-b83b-6e8df2c43e5acode_foldedäcode%hide_solution(
md"""
We use variables ``S_1 \in \{\text{t},\text{f}\}`` and ``S_2 \in \{\text{y},\text{n}\}`` for statements 1 and 2 and shorthand "y", "n", "t" and "f" for "yes", "no", "true" and "false", respectively. The problem statement provides us with the following probabilities,
```math
\begin{align}
p(S_1=\text{t}) &= 1/3 \\
p(S_1=\text{f}) &= 1 - p(S_1=\text{t}) = 2/3\\
p(S_2=\text{y} | S_1=\text{t}) &= 1/3 \\
p(S_2=\text{y} | S_1=\text{f}) &= 2/3
\end{align}
```
We are asked to compute ``p(S_1=\text{t} | S_2=\text{y})``. Use Bayes rule,
```math
\begin{align}
p(S_1=\text{t} | S_2=\text{y}) &= \frac{p(S_1=\text{t},S_2=\text{y})}{p(S_2=\text{y})} \\
&= \frac{\overbrace{p(S_2=\text{y}|S_1=\text{t})p(S_1=\text{t})}^{\text{both speak the truth}}}{\underbrace{p(S_2=\text{y}|S_1=\text{t})p(S_1=\text{t})}_{\text{both speak the truth}}+\underbrace{p(S_2=\text{y}|S_1=\text{f})p(S_1=\text{f})}_{\text{both lie}}}\\
&= \frac{\frac{1}{3}\cdot\frac{1}{3}}{\frac{1}{3}\cdot\frac{1}{3}+\frac{2}{3}\cdot\frac{2}{3}} = \frac{1}{5}
\end{align}
```
""")$3e1babca-d294-11ef-37c1-cd821a6488b2metadatadisabled©show_logsîskip_as_script§cell_id$3e1babca-d294-11ef-37c1-cd821a6488b2code_foldedäcodemd"""
Note that marginalization can be understood as applying a "generalized" sum rule. Bishop (p.14) and some other authors also refer to this as the sum rule, but we do not follow that terminology.
"""$58f70d3e-4b64-414e-b560-327be2a0c4c2metadatadisabled©show_logsîskip_as_script§cell_id$58f70d3e-4b64-414e-b560-327be2a0c4c2code_foldedäcode:exercise_statement("The PDF for the Sum of Two Variables")$34f1f3cb-9346-4bd3-ae9f-a25b4d5de781metadatadisabled©show_logsîskip_as_script§cell_id$34f1f3cb-9346-4bd3-ae9f-a25b4d5de781code_folded¤codeIz = Normal(
x.μ + y.μ, # mean
√(x.σ^2 + y.σ^2) # stdev
)$3e18c25c-d294-11ef-11bc-a93c2572b107metadatadisabled©show_logsîskip_as_script§cell_id$3e18c25c-d294-11ef-11bc-a93c2572b107code_foldedäcodemd"""
In developing this calculus, only some very agreeable assumptions were made, including:
- Degrees of belief are represented by real numbers.
- Plausibility assessments are consistent, e.g.,
- if ``A`` becomes more plausible under new information ``B``, the assigned degree-of-belief should increase accordingly.
- If the belief in ``A`` is greater than the belief in ``B``, and the belief in ``B`` is greater than the belief in ``C``, then the belief in ``A`` must be greater than the belief in ``C``.
- Logical equivalences are preserved, e.g.,
- If the belief in an event can be inferred in two different ways, for example, either by first updating on information ``I_1`` and then ``I_2`` or the other way around, then the two ways must agree on the resulting belief.
"""$03692f4d-0daf-4dfc-a7ff-6b954326e4d0metadatadisabled©show_logsîskip_as_script§cell_id$03692f4d-0daf-4dfc-a7ff-6b954326e4d0code_foldedäcodeexercises()
$3e1e4dda-d294-11ef-33b7-4bbe3300ca22metadatadisabled©show_logsîskip_as_script§cell_id$3e1e4dda-d294-11ef-33b7-4bbe3300ca22code_foldedäcodeUmd"""
## Moments of the PDF
Distributions can often usefully be summarized by a set of values known as moments of the distribution.
Consider a distribution ``p(x)``. The first moment, also known as **expected value** or **mean** of ``p(x)`` is defined as
```math
\mu_x = \mathbb{E}[x] \triangleq \int x \,p(x) \,\mathrm{d}{x}
```
"""$3e1d33c8-d294-11ef-0a08-bdc419949925metadatadisabled©show_logsîskip_as_script§cell_id$3e1d33c8-d294-11ef-0a08-bdc419949925code_foldedäcodeUmd"""
## Probabilistic Inference
**Probabilistic inference** refers to computing
```math
p(\,\text{whatever-we-want-to-know}\, | \,\text{whatever-we-already-know}\,)
```
For example:
```math
\begin{align*}
p(\,\text{Mr.S.-killed-Mrs.S.} \;&|\; \text{he-has-her-blood-on-his-shirt}\,) \\
p(\,\text{transmitted-codeword} \;&|\;\text{received-codeword}\,)
\end{align*}
```
This can be accomplished by repeatedly applying the sum and product rules.
In particular, consider a joint distribution ``p(X,Y,Z)``. Assume we are interested in ``p(X|Z)``:
```math
\begin{align*}
p(X|Z) \stackrel{p}{=} \frac{p(X,Z)}{p(Z)} \stackrel{s}{=} \frac{\sum_Y p(X,Y,Z)}{\sum_{X,Y} p(X,Y,Z)} \,,
\end{align*}
```
where the ``s`` and ``p`` above the equality sign indicate whether the sum or product rule was used.
In the rest of this course, we'll encounter many lengthy probabilistic derivations. For each manipulation, you should be able to associate an 's' (for sum rule), a 'p' (for product or Bayes rule) or an 'm' (for a simplifying model assumption like conditional independency) above any equality sign.
"""$3e194ef2-d294-11ef-3b38-1ddc3063ff35metadatadisabled©show_logsîskip_as_script§cell_id$3e194ef2-d294-11ef-3b38-1ddc3063ff35code_foldedäcodemd"""
The Bayesian interpretation contrasts with the **frequentist** interpretation of a probability as the relative frequency that an event would occur under repeated execution of an experiment.
* For instance, if the experiment is tossing a coin, then ``p(\texttt{outcome} = \texttt{tail}) = 0.4`` means that in the limit of a large number of coin tosses, 40% of outcomes turn up as ``\texttt{tail}``.
"""$16c2eb59-16b8-4347-9aab-6e4b99016c79metadatadisabled©show_logsîskip_as_script§cell_id$16c2eb59-16b8-4347-9aab-6e4b99016c79code_foldedäcodeNkeyconcept("", md"Bayes rule is the fundamental rule for learning from data!")$3e1906ea-d294-11ef-236e-c966a9474170metadatadisabled©show_logsîskip_as_script§cell_id$3e1906ea-d294-11ef-236e-c966a9474170code_foldedäcode_md"""
In general, nearly all interesting questions in machine learning (and information processing in general) can be stated in the following form (a conditional probability):
```math
p(\texttt{whatever-we-want-to-know}\, | \,\texttt{whatever-we-do-know})
```
where ``p(A|B)`` means the probability that ``A`` is true, given that ``B`` is true.
"""$3e1bcb00-d294-11ef-2795-bd225bd00496metadatadisabled©show_logsîskip_as_script§cell_id$3e1bcb00-d294-11ef-2795-bd225bd00496code_foldedäcodermd"""
## $(HTML("
Bayes Rule "))
Consider two variables ``D`` and ``\theta``. It follows from symmetry arguments that
```math
p(D,\theta)=p(\theta,D)\,,
```
and hence that
```math
p(D|\theta)p(\theta)=p(\theta|D)p(D)\,,
```
or equivalently,
```math
p(\theta|D) = \frac{p(D|\theta) }{p(D)}p(\theta)\,.\qquad \text{(Bayes rule)}
```
"""$66ebe33c-8360-4938-9b51-625e5bed176cmetadatadisabled©show_logsîskip_as_script§cell_id$66ebe33c-8360-4938-9b51-625e5bed176ccode_foldedäcodehide_solution(
md"""
That depends. The term “probabilistic” refers to a state-of-knowledge (or beliefs) about something—in this case, about the values of a speech signal. The key point is that the signal itself is neither inherently probabilistic nor deterministic; these labels describe our knowledge about it.
If you had a perfect microphone and recorded the speech signal flawlessly at its source, you would know all its values exactly—no uncertainty—so you could call it deterministic.
However, before making the recording, how would you represent your knowledge about the signal values you are going to measure? You face uncertainty, so the appropriate description is a probability distribution over all possible signal values.
""")$3e1e9224-d294-11ef-38b3-137c2be22400metadatadisabled©show_logsîskip_as_script§cell_id$3e1e9224-d294-11ef-38b3-137c2be22400code_foldedäcodekmd"""
## Linear Transformations
Consider an arbitrary distribution ``p(x)`` with mean ``\mu_x`` and covariance matrix ``\Sigma_x``. Define
```math
z = A x + b \,.
```
No matter the specification of ``p(x)``, the mean and covariance matrix for ``z`` are given by
```math
\begin{align*}
\mu_z &= A\mu_x + b \\
\Sigma_z &= A\,\Sigma_x\,A^T
\end{align*}
```
"""$3e17df5e-d294-11ef-38c7-f573724871d8metadatadisabled©show_logsîskip_as_script§cell_id$3e17df5e-d294-11ef-38c7-f573724871d8code_foldedäcode"title("Probability Theory Review")$70d79732-0f55-40ba-929d-fba431318848metadatadisabled©show_logsîskip_as_script§cell_id$70d79732-0f55-40ba-929d-fba431318848code_foldedäcode0md"""
##### Disease diagnosis implementation
"""$3e1964b4-d294-11ef-373d-712257fc130fmetadatadisabled©show_logsîskip_as_script§cell_id$3e1964b4-d294-11ef-373d-712257fc130fcode_foldedäcodemd"""
The Bayesian viewpoint is more generally applicable than the frequentist viewpoint, e.g., it is hard to apply the frequentist viewpoint to events like ``\texttt{"it will rain tomorrow"}``.
"""$3e1c9184-d294-11ef-3e35-5393d97fbc44metadatadisabled©show_logsîskip_as_script§cell_id$3e1c9184-d294-11ef-3e35-5393d97fbc44code_foldedäcodeنmd"""
Note that ``L(\theta)`` is not a probability distribution for ``\theta`` since in general ``\sum_\theta L(\theta) \neq 1``.
"""$3e18f18c-d294-11ef-33e4-b7f9495e0508metadatadisabled©show_logsîskip_as_script§cell_id$3e18f18c-d294-11ef-33e4-b7f9495e0508code_foldedäcode md"""
## Why Probability Theory for Machine Learning?
Machine learning concerns updating our beliefs about appropriate settings for model parameters from new information (namely a data set), and therefore PT provides the *optimal calculus for machine learning*.
"""$3e1b5c9c-d294-11ef-137f-d75b3731eae4metadatadisabled©show_logsîskip_as_script§cell_id$3e1b5c9c-d294-11ef-137f-d75b3731eae4code_foldedäcodemd"""
We mentioned before that every inference problem in PT can be evaluated through the sum and product rules. Next, we present two useful corollaries:
- (1) *Marginalization*
- (2) *Bayes rule*.
"""$4edf38ab-a940-4ab0-be22-fa95cf571146metadatadisabled©show_logsîskip_as_script§cell_id$4edf38ab-a940-4ab0-be22-fa95cf571146code_foldedäcodemd"""
In the Bayesian interpretation, all probabilities are, in principle, conditional probabilities of the type ``p(A|I)``, since there is always some background knowledge. However, we often write ``p(A)`` rather than ``p(A|I)`` if the background knowledge ``I`` is assumed to be obviously present. E.g., we usually write ``p(A)`` rather than ``p(\,A\,|\,\text{the-sun-comes-up-tomorrow}\,)``.
"""$a66ab9df-897c-42e5-8b0f-c520ceaffa23metadatadisabled©show_logsîskip_as_script§cell_id$a66ab9df-897c-42e5-8b0f-c520ceaffa23code_folded¤codeusing Printf$3e19e95a-d294-11ef-3da4-6d23922a5150metadatadisabled©show_logsîskip_as_script§cell_id$3e19e95a-d294-11ef-3da4-6d23922a5150code_foldedäcode4md"""
## Variable Assignments as Propositions
"""$3e19436c-d294-11ef-11c5-f9914f7a3a57metadatadisabled©show_logsîskip_as_script§cell_id$3e19436c-d294-11ef-11c5-f9914f7a3a57code_foldedäcode
md"""
In the **Bayesian** interpretation, the probability is associated with a **state-of-knowledge** (usually held by a person, but formally by a rational agent).
* For instance, in a coin tossing experiment, ``p(\texttt{outcome} = \texttt{tail}) = 0.4`` should be interpreted as the belief that there is a 40% chance that ``\texttt{tail}`` comes up if the coin were tossed.
* Under the Bayesian interpretation, PT calculus (sum and product rules) **extends boolean logic to rational reasoning with uncertainty**.
"""$3e1b05ee-d294-11ef-33de-efed64d01c0dmetadatadisabled©show_logsîskip_as_script§cell_id$3e1b05ee-d294-11ef-33de-efed64d01c0dcode_foldedäcode[keyconcept(
"",
md"""
All valid probabilistic relations can be derived from just two fundamental principles: the **sum rule** and the **product rule**. These two rules form the foundation of probability theory, from which more complex constructs such as conditional probabilities, Bayes’ theorem, and marginalization naturally follow.
"""
)$4f6dd225-c64d-4b76-b075-0bf71c863b5ametadatadisabled©show_logsîskip_as_script§cell_id$4f6dd225-c64d-4b76-b075-0bf71c863b5acode_folded¤code
begin
prevalence_bond = @bind prevalence Scrubbable(0:0.01:1; format=".0%", default=0.01)
sensitivity_bond = @bind sensitivity Scrubbable(0:0.01:1; format=".0%", default=0.95)
specificity_bond = @bind specificity Scrubbable(0:0.01:1; format=".0%", default=0.85)
end;$61713e1c-8e37-45d9-9f58-c3db69e15b66metadatadisabled©show_logsîskip_as_script§cell_id$61713e1c-8e37-45d9-9f58-c3db69e15b66code_foldedäcode7challenge_statement("Disease Diagnosis",header_level=1)$3e1f7d5e-d294-11ef-2878-05744036f32cmetadatadisabled©show_logsîskip_as_script§cell_id$3e1f7d5e-d294-11ef-2878-05744036f32ccode_foldedäcodemd"""
## General Variable Transformations
Suppose ``x`` is a **discrete** variable with probability **mass** function ``P_x(x)``, and ``y = h(x)`` is a one-to-one function with ``x = g(y) = h^{-1}(y)``. Then
```math
P_y(y) = P_x(g(y))\,.
```
"""$3e1b7d14-d294-11ef-0d10-1148a928dd57metadatadisabled©show_logsîskip_as_script§cell_id$3e1b7d14-d294-11ef-0d10-1148a928dd57code_foldedäcodeLmd"""
## Marginalization (through the Law of Total Probability)
Let ``A`` and ``B_1,B_2,\ldots,B_n`` be events, where ``B_1,B_2,\ldots,B_n`` partitions the universe. Then
```math
\sum_{i=1}^n p(A,B_i) = p(A) \,.
```
This rule is called the [law of total probability](https://en.wikipedia.org/wiki/Law_of_total_probability).
"""$2292c667-862e-4924-8f15-de0a8893c439metadatadisabled©show_logsîskip_as_script§cell_id$2292c667-862e-4924-8f15-de0a8893c439code_foldedäcodeGhide_solution(
md"""
```math
\begin{align*}
\Sigma_{yx}^T &\triangleq \Big( \mathbb{E}\left[ (y-\mu_y)(x-\mu_x)^T \right] \Big)^T \\
&= \mathbb{E}\left[ \big((y-\mu_y)(x-\mu_x)^T \big)^T\right] \\
&= \mathbb{E}\left[ (x-\mu_x)(y-\mu_y)^T \right] \quad \text{using }(AB)^T = B^TA^T \\
&= \Sigma_{xy}
\end{align*}
```
""" )$3e1ed1a4-d294-11ef-2de4-d7cc540e06a1metadatadisabled©show_logsîskip_as_script§cell_id$3e1ed1a4-d294-11ef-2de4-d7cc540e06a1code_foldedäcodewmd"""
More generally, assume two jointly continuous variables ``X`` and ``Y``, with joint PDF ``p_{xy}(x,y)``. Let ``Z=X+Y``, then
```math
\begin{align*}
\text{Prob}(Z\leq z) &= \text{Prob}(X+Y\leq z)\\
&= \int_{-\infty}^\infty \biggl( \int_{-\infty}^{z-x} p_{xy}(x,y) \mathrm{d}y \biggr) \mathrm{d}x \\
&= \int_{-\infty}^\infty \biggl( \int_{-\infty}^{z} p_{xy}(x,t-x) \mathrm{d}t \biggr) \mathrm{d}x \\
&= \int_{-\infty}^z \biggl( \underbrace{\int_{-\infty}^{\infty} p_{xy}(x,t-x) \mathrm{d}x}_{p_z(t)} \biggr) \mathrm{d}t
\end{align*}
```
Hence, the PDF for the sum ``Z`` is given by ``p_z(z) = \int_{-\infty}^{\infty} p_{xy}(x,z-x) \mathrm{d}x``.
In particular, if ``X`` and ``Y`` are **independent** variables, then
```math
p_z (z) = \int_{-\infty}^{\infty} p_x(x) p_y(z - x)\,\mathrm{d}{x} = p_x(z) * p_y(z)\,,
```
which is the **convolution** of the two marginal PDFs.
"""$dd31ec7c-708d-4fd7-958d-f9887798a5bcmetadatadisabled©show_logsîskip_as_script§cell_id$dd31ec7c-708d-4fd7-958d-f9887798a5bccode_foldedäcodemd"""
# Code
"""$577c6e08-9b74-43a4-9e27-9c477980406bmetadatadisabled©show_logsîskip_as_script§cell_id$577c6e08-9b74-43a4-9e27-9c477980406bcode_foldedäcodeLfunction NormalBind(bond_mu, bond_sigma; style_css::String="")
PlutoUI.Experimental.transformed_value(
function((x,y))
Normal(x,y)
end,
PlutoUI.combine() do Child
@htl """
Normal(μ=$(Child(bond_mu)), σ=$(Child(bond_sigma)))
"""
end
)
end$7d493e09-f7cc-4e13-a506-b792edcbf390metadatadisabled©show_logsîskip_as_script§cell_id$7d493e09-f7cc-4e13-a506-b792edcbf390code_foldedäcodeemd"""
and the likelihood function
```math
L(\theta) \triangleq p(y=1|\theta) = \theta \,.
```
"""$727dc817-0284-4c0f-9a92-21dcbea50807metadatadisabled©show_logsîskip_as_script§cell_id$727dc817-0284-4c0f-9a92-21dcbea50807code_foldedäcode*hide_solution(
md"""
(a) ``p(S_1=R) = \frac{N_\text{red}}{N_\text{red}+N_\text{green}} = \frac{5}{12}``
(b) The outcome of the ``n``-th draw is referred to by ``S_n``. Use Bayes rule to get,
```math
\begin{align}
p(S_1=\text{R} | S_2=\text{G}) &= \frac{p(S_2=\text{G} | S_1=\text{R}) p(S_1=\text{R})} {p(S_2=\text{G} | S_1 = \text{R}) p(S_1=\text{R}) + p(S_2=\text{G} | S_1=\text{G}) p(S_1=\text{G})} \\
&= \frac{\frac{7}{11}\cdot\frac{5}{12}}{\frac{7}{11}\cdot\frac{5}{12}+\frac{6}{11}\cdot\frac{7}{12}} \\
&= \frac{5}{11}
\end{align}
```
""")$3e1e2b96-d294-11ef-3a68-fdc78232142emetadatadisabled©show_logsîskip_as_script§cell_id$3e1e2b96-d294-11ef-3a68-fdc78232142ecode_foldedäcodecmd"""
##### Problem
- A dark bag contains five red balls and seven green ones.
- (a) What is the probability of drawing a red ball on the first draw?
- Balls are not returned to the bag after each draw.
- (b) If you know that on the second draw the ball was a green one, what is now the probability of drawing a red ball on the first draw?
"""$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4metadatadisabled©show_logsîskip_as_script§cell_id$4c639e65-e06b-4c5e-b6e7-aabed6b6c0b4code_foldedäcodelhide_solution(
md"""
There are two hypotheses: let ``H = 0`` mean that the original ball in the bag was white and ``H = 1`` that it was black. Assume the prior probabilities are equal, i.e.,
```math
P(H =0) = 1/2, \quad P(H =1) = 1/2 \,.
```
The data is that when a randomly selected ball was drawn from the bag, which contained a white one and the unknown one, it turned out to be white. The probability of this result according to each hypothesis is:
```math
P(D|H =0) = 1, \quad P(D|H =1) = 1/2 \,.
```
So by Bayes theorem,
```math
\begin{align}
P(H=0|D) &= \frac{P(H=0,D)}{P(D)} \\
&= \frac{P(D|H=0) P(H=0)}{P(D|H=0) P(H=0) + P(D|H=1) P(H=1)} \\
&= \frac{1 \cdot \frac{1}{2}}{1 \cdot \frac{1}{2} + \frac{1}{2} \cdot \frac{1}{2}} \\
&= \frac{2}{3}
\end{align}
```
and consequently,
```math
P(H =1|D) = 1 - P(H =0|D) = \frac{1}{3}
```
""")$3e18b2fa-d294-11ef-1255-df048f0dcec2metadatadisabled©show_logsîskip_as_script§cell_id$3e18b2fa-d294-11ef-1255-df048f0dcec2code_foldedäcodemd"""
[Richard T. Cox (1946)](https://aapt.scitation.org/doi/10.1119/1.1990764) developed a **calculus for rational reasoning** about how to represent and update the **degree-of-belief** about the truth value of an event when faced with new information.
"""$3e185ab0-d294-11ef-3f7d-9bd465518274metadatadisabled©show_logsîskip_as_script§cell_id$3e185ab0-d294-11ef-3f7d-9bd465518274code_foldedäcodemd"""
##### Problem
- Given is a disease with a prevalence of $(prevalence_bond) and a test procedure with sensitivity ('true positive' rate) of $(sensitivity_bond), and specificity ('true negative' rate) of $(specificity_bond). What is the chance that somebody who tests positive actually has the disease?
##### Solution
- [See later in this lecture](#Challenge-Revisited:-Disease-Diagnosis). We will use probabilistic inference, which we will discuss next.
"""$3e1c0e80-d294-11ef-0d19-375e01988f16metadatadisabled©show_logsîskip_as_script§cell_id$3e1c0e80-d294-11ef-0d19-375e01988f16code_foldedäcodemd"""
Note that the evidence (a.k.a. *marginal likelihood* ) can be computed from the numerator through marginalization since
```math
p(D) = \int p(D,\theta) \,\mathrm{d}\theta = \int p(D|\theta)\,p(\theta) \,\mathrm{d}\theta
```
"""$5da42e9a-4318-48ea-9f43-4c1e1c97bcebmetadatadisabled©show_logsîskip_as_script§cell_id$5da42e9a-4318-48ea-9f43-4c1e1c97bcebcode_foldedäcodekeyconceptsummary()$a8d4a517-84a7-426e-a49e-482c5fd047aemetadatadisabled©show_logsîskip_as_script§cell_id$a8d4a517-84a7-426e-a49e-482c5fd047aecode_foldedäcodemd"""
##### The Likelihood Function is a Function of What? (*)
When considering the distribution ``p(D|\theta)``, is it more correct to speak about the likelihood of the model parameters ``\theta`` than about the likelihood of the observed data set ``D``. And why?
"""$3e196d6a-d294-11ef-0795-41c045079251metadatadisabled©show_logsîskip_as_script§cell_id$3e196d6a-d294-11ef-0795-41c045079251code_foldedäcodeٝmd"""
The Bayesian viewpoint is clearly favored in the machine learning community. (In this class, we also strongly favor the Bayesian interpretation).
"""$218d3b6e-50b6-4b98-a00c-a19dd33d2c03metadatadisabled©show_logsîskip_as_script§cell_id$218d3b6e-50b6-4b98-a00c-a19dd33d2c03code_foldedäcodeCmd"""
Let's plot the distributions for ``X``, ``Y``, and ``Z``.
"""$3e1b4b1c-d294-11ef-0423-9152887cc403metadatadisabled©show_logsîskip_as_script§cell_id$3e1b4b1c-d294-11ef-0423-9152887cc403code_foldedäcodemd"""
## Independent, Exclusive, and Exhaustive Events
It will be helpful to introduce some terms concerning special relationships between events.
##### Joint events
The expression ``p(A,B)`` for the probability of the conjuction ``A \land B`` is also called the **joint probability** of events ``A`` and ``B``. Note that
```math
p(A,B) = p(B,A)\,,
```
since ``A\land B = B \land A``. Therefore, the order of arguments in a joint probability distribution does not matter: ``p(A,B,C,D) = p(C,A,D,B)``, etc.
##### Independent events
Two events ``A`` and ``B`` are said to be **independent** if the probability of one event is not altered by information about the truth of the other event, i.e.,
```math
p(A|B) = p(A)\,.
```
It follows that, if ``A`` and ``B`` are independent, then the product rule simplifies to
```math
p(A,B) = p(A) p(B)\,.
```
``A`` and ``B`` with given background ``I`` are said to be **conditionally independent** for given ``I``, if
```math
p(A|B,I) = p(A|I)\,.
```
In that case, the product rule simplifies to ``p(A,B|I) = p(A|I) p(B|I)``.
##### Mutually exclusive events
Two events ``A_1`` and ``A_2`` are said to be **mutually exclusive** ('disjoint') if they cannot be true simultaneously, i.e., if
```math
p(A_1,A_2)=0 \,.
```
For mutually exclusive events, probabilities add (this follows from the sum rule), hence
```math
p(A_1 + A_2) = p(A_1) + p(A_2)
```
##### Collectively exhaustive events
A set of events ``A_1, A_2, \ldots, A_N`` is said to be **collectively exhaustive** if one of the statements is necessarily true, i.e., ``A_1+A_2+\cdots +A_N=\mathrm{TRUE}``, or equivalently
```math
p(A_1+A_2+\cdots +A_N) = 1 \,.
```
##### Partitioning the universe
If a set of events ``A_1, A_2, \ldots, A_n`` is both **mutually exclusive** and **collectively exhaustive**, then we say that they **partition the universe**. Technically, this means that
```math
\sum_{n=1}^N p(A_n) = p(A_1 + \ldots + A_N) = 1
```
"""$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52cmetadatadisabled©show_logsîskip_as_script§cell_id$d1ab6fc4-e92c-4e77-8c4a-f5f33cb0b52ccode_foldedäcode@htl """
Parameters for X
$(@bind x NormalBind(Scrubbable(2.0), Scrubbable(1.0)))
Parameters for Y
$(@bind y NormalBind(Scrubbable(2.0), Scrubbable(0.5)))
"""$3e1eeb14-d294-11ef-1702-f5d2cf6fe60ametadatadisabled©show_logsîskip_as_script§cell_id$3e1eeb14-d294-11ef-1702-f5d2cf6fe60acode_foldedäcodemd"""
[Wikipedia's List of convolutions of probability distributions](https://en.wikipedia.org/wiki/List_of_convolutions_of_probability_distributions) shows how these convolutions work out for a few common probability distributions.
"""$922770f4-ddc8-4089-b378-f14088276b43metadatadisabled©show_logsîskip_as_script§cell_id$922770f4-ddc8-4089-b378-f14088276b43code_foldedäcodeJexercise_statement("Which color does the ball have?"; prefix="Inference ")$f8c8ba53-df36-48a6-afde-2952cbcfbe48metadatadisabled©show_logsîskip_as_script§cell_id$f8c8ba53-df36-48a6-afde-2952cbcfbe48code_foldedäcodemd"""
Does this fragment resonate with your own experience?
In this lesson we introduce *Probability Theory* (PT) again. As we will see in the next lessons, PT is all you need to make sense of machine learning, artificial intelligence, statistics, etc.
"""$41bee964-a0a9-4a7f-8505-54a9ee12ef0dmetadatadisabled©show_logsîskip_as_script§cell_id$41bee964-a0a9-4a7f-8505-54a9ee12ef0dcode_foldedäcodemd"""
## Propositional (Boolean) Logic
Define an **event** (or "proposition") ``A`` as a statement that can be considered for its truth by a person. For instance,
```math
𝐴= \texttt{``there is life on Mars''}
```
Boolean logic (or propositional logic) is a formal system of logic based on binary truth values: every proposition is either true (with assigned value ``1``) or false (with assigned value ``0``). It is named after George Boole, who developed the algebraic formulation of logic in the mid-19th century.
With Boolean operators (``\lor``, ``\land``, ``\implies``, etc.), we can create and evaluate compound propositions, e.g.,
- Given two events ``A`` and ``B``, the **conjunction** (logical-and) ``A \land B`` is true if-and-only-if both ``A`` and ``B`` are true. We write ``A \land B`` also shortly as ``AB`` (or use a comma as in a joint probability distribution ``p(A,B)``).
- The **disjunction** (logical-or) ``A \lor B``, is true if either ``A`` or ``B`` is true or both ``A`` and ``B`` are true. We write ``A \lor B`` also as ``A + B`` (Note that the plus-sign is here not an arithmetic operator, but rather a logical operator to process truth values.)
- The denial of ``A``, i.e., the event **not**-A, is written as ``\bar{A}``.
Boolean logic provides the rules of inference for **deductive reasoning** and underpins all formal reasoning systems in mathematics and philosophy.
"""$3e1b8bf4-d294-11ef-04cc-6364e46fdd64metadatadisabled©show_logsîskip_as_script§cell_id$3e1b8bf4-d294-11ef-04cc-6364e46fdd64code_foldedäcodemd"""
A very practical application of this law is to get rid of a variable that we are not interested in. For instance, if ``X`` and ``Y \in \{y_1,y_2,\ldots,y_n\}`` are discrete variables, then
```math
p(X) = \sum_{i=1}^n p(X,Y=y_i)\,.
```
"""$98fa17a6-7c8b-46e4-b32d-52db183d88f8metadatadisabled©show_logsîskip_as_script§cell_id$98fa17a6-7c8b-46e4-b32d-52db183d88f8code_foldedäcodeFmd"""
Set the parameters for the distributions of ``X`` and ``Y``:
"""$3e1f4f46-d294-11ef-29b8-69e546763781metadatadisabled©show_logsîskip_as_script§cell_id$3e1f4f46-d294-11ef-29b8-69e546763781code_foldedäcodemd"""
## PDF for the Product of Two Variables
For two continuous **independent** variables ``X`` and ``Y``, with PDF's ``p_x(x)`` and ``p_y(y)``, the PDF of ``Z = X Y`` is given by
```math
p_z(z) = \int_{-\infty}^{\infty} p_x(x) \,p_y(z/x)\, \frac{1}{|x|}\,\mathrm{d}x\,.
```
For proof, see [https://en.wikipedia.org/wiki/Product_distribution](https://en.wikipedia.org/wiki/Product_distribution).
"""$3a1d380e-df80-4727-9772-f199214cf05dmetadatadisabled©show_logsîskip_as_script§cell_id$3a1d380e-df80-4727-9772-f199214cf05dcode_foldedäcodeټmd"""
##### The Sum Rule (**)
Derive the general sum rule,
```math
p(A + B) = p(A) + p(B) - p(A,B)
```
from the _elementary_ sum rule ``p(A) + p(\bar A) = 1`` and the product rule.
"""$3e1e5a5a-d294-11ef-2fdf-efee4eb1a0f2metadatadisabled©show_logsîskip_as_script§cell_id$3e1e5a5a-d294-11ef-2fdf-efee4eb1a0f2code_foldedäcode٧md"""
The second central moment, also known as **variance** of ``x`` is defined as
```math
\Sigma_x \triangleq \mathbb{E} \left[(x-\mu_x)(x-\mu_x)^T \right]
```
"""$3e1bdd02-d294-11ef-19e8-2f44eccf58afmetadatadisabled©show_logsîskip_as_script§cell_id$3e1bdd02-d294-11ef-19e8-2f44eccf58afcode_foldedäcodefmd"""
This last formula is called **Bayes rule**, named after its inventor [Thomas Bayes](https://en.wikipedia.org/wiki/Thomas_Bayes) (1701-1761). While Bayes rule is always true, a particularly useful application occurs when ``D`` refers to an observed data set and ``\theta`` is set of unobserved model parameters. In that case,
* the **prior** probability ``p(\theta)`` represents our **state-of-knowledge** about proper values for ``\theta``, before seeing the data ``D``.
* the **posterior** probability ``p(\theta|D)`` represents our state-of-knowledge about ``\theta`` after we have seen the data.
"""$fea8ae4c-8ef9-4b74-ad13-1314afef97demetadatadisabled©show_logsîskip_as_script§cell_id$fea8ae4c-8ef9-4b74-ad13-1314afef97decode_foldedäcodemd"""
## True and False Events
In probability theory, events that are certainly true or certainly false are treated as special cases of general events, and they correspond to probabilities of ``1`` and ``0``, respectively.
Let ``\Omega`` be the sample space, i.e., the set of all possible outcomes of an experiment. Then:
- The true event corresponds to the entire sample space ``\Omega``.
- It always happens, regardless of the outcome.
- For example, for the outcome of a throw of a dice, the sample space is ``\Omega = \{1,2,3,4,5,6\}``.
- Its probability is
```math
p(\Omega) = 1\,.
```
The false event corresponds to the empty set ``\emptyset``.
- It never happens (no possible outcomes).
- Its probability is
```math
p(\emptyset) = 0 \,.
```
"""$db73766d-643c-41d7-a1eb-f376c657f860metadatadisabled©show_logsîskip_as_script§cell_id$db73766d-643c-41d7-a1eb-f376c657f860code_foldedäcodemd"""
If the transformation ``y=h(x)`` is not invertible, then ``x=g(y)`` does not exist. In that case, you can still work out the transformation by equating equivalent probability masses in the two domains.
"""$3b1b0869-b815-4697-9dba-3c4b4cb5ac47metadatadisabled©show_logsîskip_as_script§cell_id$3b1b0869-b815-4697-9dba-3c4b4cb5ac47code_foldedäcode.hide_solution(
md"""
```math
\begin{align}
p\left( A + B \right) &\underset{\mathrm{bool}}{=} p\left( \overline {\bar A \bar B } \right) \\
&\underset{\mathrm{sum}}{=} 1 - p\left( \bar{A} \bar{B} \right) \\
&\underset{\mathrm{prod}}{=} 1 - p\left( \bar{A} |\bar{B} \right) p\left(\bar{B} \right) \\
&\underset{\mathrm{sum}}{=} 1 - \left( 1 - p\left(A|\bar B \right) \right) \left( 1 - p\left( B \right) \right) \\
&= p(B) + \left( {1 - p\left( B \right)} \right)p\left( {A|\bar B } \right) \\
&\underset{\mathrm{prod}}{=} p(B) + \left( 1 - p\left( B \right) \right) p\left( \bar{B} |A \right) \frac{ p\left( A \right) }{ p\left(\bar{B}\right)} \\
&\underset{\mathrm{sum}}{=} p(B) + p\left(\bar{B} |A \right) p\left( A \right) \\
&\underset{\mathrm{sum}}{=} p(B) + \left( 1 - p\left( {B|A} \right) \right) p\left( A \right) \\
&\underset{\mathrm{sum}}{=} p\left( A \right) + p(B) - p\left( A,B \right)
\end{align}
```
Note that, aside from the first boolean rewrite, everything follows straight application of sum and product rules.
""")$a8046381-ff11-40af-ae2b-078d71c586e7metadatadisabled©show_logsîskip_as_script§cell_id$a8046381-ff11-40af-ae2b-078d71c586e7code_foldedäcodeeresult = (sensitivity * prevalence) / (sensitivity * prevalence + (1 - specificity)*(1 - prevalence))$3e1d20e0-d294-11ef-2044-e1fe6590a600metadatadisabled©show_logsîskip_as_script§cell_id$3e1d20e0-d294-11ef-2044-e1fe6590a600code_foldedäcodemd"""
!!! note
The (discrete) sampling distribution is a valid probability distribution.
However, the likelihood function ``L(\theta)`` clearly isn't, since ``\int_0^1 L(\theta) \mathrm{d}\theta = 0.5 \neq 1``.
"""$fc733d61-fd0f-4a13-9afc-4505ac0253dfmetadatadisabled©show_logsîskip_as_script§cell_id$fc733d61-fd0f-4a13-9afc-4505ac0253dfcode_foldedäcodewbegin
f(y,θ) = θ.^y .* (1 .- θ).^(1 .- y); # p(y|θ)
θ_bond = @bind θ Scrubbable(0.0:0.02:1; format=".2f");
end;$3e1c60ba-d294-11ef-3a01-cf9e97512857metadatadisabled©show_logsîskip_as_script§cell_id$3e1c60ba-d294-11ef-3a01-cf9e97512857code_foldedäcodemd"""
In general, ``p(D|\theta)`` is just a function of the two variables ``D`` and ``\theta``. We distinguish two interpretations of this function, depending on which variable is observed (or given by other means).
"""$3e1bf116-d294-11ef-148b-f7a1ca3f3badmetadatadisabled©show_logsîskip_as_script§cell_id$3e1bf116-d294-11ef-148b-f7a1ca3f3badcode_foldedäcodeomd"""
Bayes rule tells us how to update our knowledge about model parameters when facing new data. Hence,
"""$3e1fa04a-d294-11ef-00c3-a51d1aaa5553metadatadisabled©show_logsîskip_as_script§cell_id$3e1fa04a-d294-11ef-00c3-a51d1aaa5553code_foldedäcodemd"""
If ``x`` is defined on a **continuous** domain, and ``p_x(x)`` is a probability **density** function, then probability mass is represented by the area under a (density) curve. In that case,
```math
p_y(y) = p_x(g(y)) g^\prime(y)\,,
```
which is also known as the [Change-of-Variable theorem](https://en.wikipedia.org/wiki/Probability_density_function#Function_of_random_variables_and_change_of_variables_in_the_probability_density_function).
"""$9b85a92b-c56b-48a3-97c1-6b1882e33a22metadatadisabled©show_logsîskip_as_script§cell_id$9b85a92b-c56b-48a3-97c1-6b1882e33a22code_foldedäcodeي@htl """
Show solution? $(@bind(show_disease_diagnosis_solution, CheckBox(default=false)))
"""$079157c9-5d97-4dbc-8c47-afa8b661db06metadatadisabled©show_logsîskip_as_script§cell_id$079157c9-5d97-4dbc-8c47-afa8b661db06code_foldedäcode٥let
θ_str = n(θ)
@mdx """
```math
p(y|\\theta=$(n(θ))) = \\begin{cases} $(n(1-θ)) & \\text{if }y=0 \\\\ $(n(θ)) & \\text{if } y=1 \\end{cases}
```
"""
end$3e1c70be-d294-11ef-14ed-0d46515541c5metadatadisabled©show_logsîskip_as_script§cell_id$3e1c70be-d294-11ef-14ed-0d46515541c5code_foldedäcode4md"""
The **sampling distribution** (a.k.a. the **data-generating** distribution)
```math
p(D|\theta=\theta_0)
```
(which is a function of ``D`` only) describes a probability distribution for data ``D``, assuming that it is generated by the given model with parameters fixed at ``\theta = \theta_0``.
"""$3e1bffec-d294-11ef-2a49-9ff0f6331addmetadatadisabled©show_logsîskip_as_script§cell_id$3e1bffec-d294-11ef-2a49-9ff0f6331addcode_foldedäcodemd"""
## Bayes Rule Nomenclature
Some nomenclature associated with Bayes rule:
```math
\underbrace{p(\theta | D)}_{\text{posterior}} = \frac{\overbrace{p(D|\theta)}^{\text{likelihood}} \times \overbrace{p(\theta)}^{\text{prior}}}{\underbrace{p(D)}_{\text{evidence}}}
```
"""$3e198ba6-d294-11ef-3fe7-d70bf4833fa6metadatadisabled©show_logsîskip_as_script§cell_id$3e198ba6-d294-11ef-3fe7-d70bf4833fa6code_foldedäcodeٕmd"""
In this class, we aim to turn on the headlights and illuminate the elegance and power of the Bayesian approach to information processing.
"""$3e1a8eca-d294-11ef-1ef0-c15b24d05990metadatadisabled©show_logsîskip_as_script§cell_id$3e1a8eca-d294-11ef-1ef0-c15b24d05990code_foldedäcodemd"""
If ``X`` is *continuously* valued, then ``p(X=x)`` is a probability *density* function (PDF) with ``p(X=x)\ge 0`` and normalization ``\int_x p(x)\mathrm{d}x=1``.
* Note that if ``X`` is continuously valued, then the value of ``p(x)`` is not necessarily ``\le 1``. E.g., a uniform distribution on the continuous domain ``[0,.5]`` has value ``p(x) = 2`` over its domain.
"""$3e1f68fa-d294-11ef-31b2-e7670da8c08cmetadatadisabled©show_logsîskip_as_script§cell_id$3e1f68fa-d294-11ef-31b2-e7670da8c08ccode_foldedäcode^md"""
Generally, this integral does not lead to an analytical expression for ``p_z(z)``.
As a crucial example, [the product of two independent variables that are both Gaussian-distributed does **not** lead to a Gaussian distribution](https://bmlip.github.io/course/minis/Sum%20and%20product%20of%20Gaussians.html).
* Exception: the distribution of the product of two variables that both have [log-normal distributions](https://en.wikipedia.org/wiki/Log-normal_distribution) is again a lognormal distribution. (If ``X`` has a normal distribution, then ``Y=\exp(X)`` has a log-normal distribution.)
"""$d0f672bd-74c9-4be1-b5ea-8d4040edb5dfmetadatadisabled©show_logsîskip_as_script§cell_id$d0f672bd-74c9-4be1-b5ea-8d4040edb5dfcode_foldedäcodeٲnavigate_prev_next(
"https://bmlip.github.io/course/lectures/Machine%20Learning%20Overview.html",
"https://bmlip.github.io/course/lectures/Bayesian%20Machine%20Learning.html"
)$3e1c51e2-d294-11ef-2c6d-d32a98308c6fmetadatadisabled©show_logsîskip_as_script§cell_id$3e1c51e2-d294-11ef-2c6d-d32a98308c6fcode_foldedäcodemd"""
## The Likelihood Function vs the Sampling Distribution
Consider a distribution ``p(D|\theta)``, where ``D`` relates to variables that are observed (i.e., a "data set") and ``\theta`` are model parameters.
"""$3e1a7c8e-d294-11ef-1f97-55e608d49141metadatadisabled©show_logsîskip_as_script§cell_id$3e1a7c8e-d294-11ef-1f97-55e608d49141code_foldedäcodeٮmd"""
If ``X`` is a *discretely* valued variable, then ``p(X=x)`` is a probability *mass* function (PMF) with ``0\le p(X=x)\le 1`` and normalization ``\sum_x p(x) =1``.
"""$b305a905-06c2-4a15-8042-72ef6375720fmetadatadisabled©show_logsîskip_as_script§cell_id$b305a905-06c2-4a15-8042-72ef6375720fcode_folded¤codeusing BmlipTeachingTools$3e1d6d00-d294-11ef-1081-e11b8397eb91metadatadisabled©show_logsîskip_as_script§cell_id$3e1d6d00-d294-11ef-1081-e11b8397eb91code_foldedäcodemd"""
##### Problem
- Given a disease ``D \in \{0, 1\}`` with prevalence (overall occurence percentage) of $(prevalence_bond) and a test procedure ``T \in \{0, 1\}`` with sensitivity (true positive rate) of $(sensitivity_bond) and specificity (true negative' rate) of $(specificity_bond), what is the chance that somebody who tests positive actually has the disease?
_The percentages are interactive! **Click and drag** to change the values._
"""$3e191b6c-d294-11ef-3174-d1b4b36e252bmetadatadisabled©show_logsîskip_as_script§cell_id$3e191b6c-d294-11ef-3174-d1b4b36e252bcode_foldedäcodevmd"""
##### Examples
* Predictions
```math
p(\,\texttt{future-observations}\,|\,\texttt{past-observations}\,)
```
* Classify a received data point ``x``
```math
p(\,x\texttt{-belongs-to-class-}k \,|\,x\,)
```
* Update a model based on a new observation
```math
p(\,\texttt{model-parameters} \,|\,\texttt{new-observation},\,\texttt{past-observations}\,)
```
"""$4a81342c-17c7-4eb9-933b-edb98df7b9c4metadatadisabled©show_logsîskip_as_script§cell_id$4a81342c-17c7-4eb9-933b-edb98df7b9c4code_folded¤code,n(x; digits=2) = @sprintf("%.*f", digits, x)$9f4125a2-d5d2-4acf-8bad-82f94af230e8metadatadisabled©show_logsîskip_as_script§cell_id$9f4125a2-d5d2-4acf-8bad-82f94af230e8code_foldedäcodeblockquote(
md"""
#### Preface
"As an undergraduate, I always found the subject of statistics to be rather mysterious. This topic wasn’t entirely new to me, as we had been taught a little bit about probability earlier at high school; for example, I was already familiar with the binomial, Poisson and normal distributions. Most of this made sense, but only seemed to relate to things like rolling dice, flipping coins, shuffling cards and so on. However, having aspirations of becoming a scientist, what I really wanted to know was how to analyse experimental data. Thus, I eagerly looked forward to the lectures on statistics. Sadly, they were a great disappointment. Although many of the tests and procedures expounded were intuitively reasonable, there was something deeply unsatisfactory about the whole affair: there didn’t seem to be any underlying basic principles! Hence, the course on ‘probability and statistics’ had led to an unfortunate dichotomy: probability made sense, but was just a game; statistics was important, but it was a bewildering collection of tests with little obvious rhyme or reason. While not happy with this situation, I decided to put aside the subject and concentrate on real science. After all, the predicament was just a reflection of my own inadequacies and I’d just have to work at it when the time came to really analyse my data.
The story above is not just my own, but is the all too common experience of many scientists. Fortunately, it doesn’t have to be like this. What we were not told in our undergraduate lectures is that there is an alternative approach to the whole subject of data analysis which uses only probability theory. In one sense, it makes the topic of statistics entirely superfluous. In another, it provides the logical justification for many of the prevalent statistical tests and procedures, making explicit the conditions and approximations implicitly assumed in their use."
""",
"D.S. Sivia"
)$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafmetadatadisabled©show_logsîskip_as_script§cell_id$840ab4dc-0d2e-4bf8-acc7-5f1ee2b0dcafcode_foldedäcode4md"""
# Probability Theory as Rational Reasoning
"""$3e027ede-4ac1-4521-a026-dc00bfca4adfmetadatadisabled©show_logsîskip_as_script§cell_id$3e027ede-4ac1-4521-a026-dc00bfca4adfcode_foldedäcode5exercise_statement("Causality?"; prefix="Inference ")$3e1889b8-d294-11ef-17bb-496655fbd618metadatadisabled©show_logsîskip_as_script§cell_id$3e1889b8-d294-11ef-17bb-496655fbd618code_foldedäcodemd"""
## The Design of Probability Theory
In the real world, we are rarely completely certain of something. Instead of a truth value for the proposition ``A``, we can attach a **degree-of-belief** ``p(A)`` that represents how likely we think ``A`` is.
Now consider the truth value of the proposition
```math
𝐴= \texttt{``there is life on Mars''}
```
with
```math
I = \texttt{``All known life forms require water''}
```
as background information. Now, assume that a new piece of information
```math
B = \texttt{``There is water on Mars''}
```
becomes available, how **should** our degree of belief in event ``A`` be affected *if we were rational*?
"""$6d07be25-53d0-46b9-b197-a3680d830952metadatadisabled©show_logsîskip_as_script§cell_id$6d07be25-53d0-46b9-b197-a3680d830952code_foldedäcodekhide_solution(
md"""
Define ``A = \begin{pmatrix} I & I \end{pmatrix}`` and ``w = \begin{pmatrix} x \\ y \end{pmatrix}``, where ``I`` is the identity matrix. Then
```math
z = A w\,.
```
Let ``\mathbb{E}[\cdot]`` refer to the expectation operator. Now apply the formula for the mean and variance of a variable after a linear transformation:
```math
\mathbb{E}[z] = \mathbb{E}[Aw] = \mathbb{E}[x+y] = \mathbb{E}[x] + \mathbb{E}[y] \,.
```
For the covariance matrix, first note that
```math
\begin{align}
\Sigma_w &= \mathbb{E} \bigg[ \begin{pmatrix} x - \mu_x \\ y- \mu_y\end{pmatrix} \begin{pmatrix} x - \mu_x \\ y- \mu_y\end{pmatrix}^T \bigg] \\
&= \begin{pmatrix} \Sigma_x & \Sigma_{xy} \\ \Sigma_{yx} & \Sigma_y\end{pmatrix}
\end{align}
```
and we note that ``\Sigma_{yx} = \Sigma_{xy}^T``. Then, ``\Sigma_z`` evaluates to
```math
\begin{align}
\Sigma_z &= A \Sigma_w A^T \\
&= \begin{pmatrix} I & I \end{pmatrix} \begin{pmatrix} \Sigma_x & \Sigma_{xy} \\ \Sigma_{yx} & \Sigma_y\end{pmatrix} \begin{pmatrix}I \\ I \end{pmatrix} \\
&= \Sigma_x + \Sigma_y + \Sigma_{xy} + \Sigma_{yx} \,.
\end{align}
```
""" )$5f377237-d9a5-4778-aa4d-1c6ce109b705metadatadisabled©show_logsîskip_as_script§cell_id$5f377237-d9a5-4778-aa4d-1c6ce109b705code_foldedäcode^md"""
##### Apples and Oranges
Box 1 contains 8 apples and 4 oranges. Box 2 contains 10 apples and 2 oranges. Boxes are chosen with equal probability. You pick a box and then select a fruit from that box.
- (a) (*) What is the probability of choosing an apple?
- (b) (**) If an apple is chosen, what is the probability that it came from box 1?
"""$d3b003c6-70ca-419f-a343-e35b266323f3metadatadisabled©show_logsîskip_as_script§cell_id$d3b003c6-70ca-419f-a343-e35b266323f3code_foldedäcodechide_solution(
md"""
Yes, it’s more correct to speak about the likelihood of the model parameters, not of the observed data set. Once ``D`` has been observed, it is no longer a random variable; it’s just a fixed outcome. What varies is ``\theta``, so ``L(\theta) = p(D|\theta)`` is a function of the parameters, not of the data.
Saying “likelihood of the data” is misleading because it confuses likelihood with the sampling distribution ``p(D|\theta)`` seen as a function of ``D`` (where ``\theta`` is fixed). The latter is a probability distribution over possible data sets before observing them.
""")$3e1fb370-d294-11ef-1fb6-63a41a024691metadatadisabled©show_logsîskip_as_script§cell_id$3e1fb370-d294-11ef-1fb6-63a41a024691code_foldedäcodemd"""
$(exercise_statement("Transformation of a Gaussian Variable"; big=true, header_level=2))
Let ``p_x(x) = \mathcal{N}(x|\mu,\sigma^2)`` where
```math
\mathcal{N}(x|\mu,\sigma^2) \triangleq \frac{1}{\sigma \sqrt{2\pi}} \,\exp\left(-\frac{(x-\mu)^2}{2 \sigma^2}\right)
```
is a [Gaussian distribution](https://bmlip.github.io/course/lectures/Density%20Estimation.html#Continuous-Data-and-the-Gaussian-Distribution). Let ``y = \frac{x-\mu}{\sigma}``.
Evaluate ``p_y(y)`` as a Gaussian distribution.
"""last_hot_reload_time shortpathProbability Theory Review.jlnbpkgbusy_packages,waiting_for_permission_but_probably_disabled§enabledðterminal_outputsMarkdownLiteralp
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...nbpkg_syncp
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...BmlipTeachingToolsp
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...LaTeXStringsp
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...Distributionsp
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...Plotsp
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...Printfp
[0m[1mWaiting for other notebooks to finish Pkg operations...[22m
[90m===[39m
[0m[1mResolving...[22m
[90m===[39m
[36m[1m Project[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Project.toml`
[36m[1m Manifest[22m[39m No packages added to or removed from `~/.julia/scratchspaces/c3e4b0f8-55cb-11ea-2926-15256bba5781/pkg_envs/env_baqychzsut/Manifest.toml`
[0m[1mInstantiating...[22m
[90m===[39m
[0m[1mPrecompiling...[22m
[90m===[39m
Waiting for notebook process to start... Done. Starting precompilation...waiting_for_permission·restart_recommended_msgrestart_required_msginstalled_versionsMarkdownLiteral0.1.5__internal_julia_version1.13.3BmlipTeachingTools1.4.1LaTeXStrings1.4.1Distributions0.25.132Plots1.41.7Printfstdlib!__internal_julia_manifest_version1.13.3install_time_nsAk+0instantiatedípluto_versionv1.1.0